Agile Data Management has become central to the art of software development and Data Governance. Technology needs to change quickly, adapting to customer and business people’s interactions in a fast-paced corporate world.
Consider a freight forwarding company in the Northwest, selling seeds to countries in Europe and Asia. The seed quantity and quality, supplied and ready for shipping, varies with the weather conditions. The commerce routes available depend on local and state politics. Should shippers strike at a port, essentially closing it down, the freight forwarder may need to find alternative transportation quickly, preventing the seed sitting in a truck for days on end and causing a loss to the shipper. The organization receiving the seed needs to do so seamlessly, even when new government tariffs and taxes appear. The freight forwarding company needs to access a data system that can adapt to these different contexts and learn about updated regulations that will impact shipping. It needs an Agile methodology approach to Data Management.
Many industries, outside of freight forwarding, realize that Data Governance needs to be Agile. From genomes to watershed data sets to policy and laws in a digital age, Data Management needs to adapt. This requirement will only grow. As noted by, Edelman and Heller’s report, “75% of marketers are concerned about threats from more agile competitors”. Companies will increase Data Analytics spending from 6.7% to 11.1 %. To see a return on investment an iterative approaches and dialogues with the customer to improve company performance will become crucial.
To help companies makes sense of Agile Data Management, the DATAVERSITY® Enterprise Data World 2017 Conference will provide sessions on Agile Data Analytics. In the meantime, this article provides a starting point in articulating Agile Data Management and its benefits, pitfalls and best practices.
What is Agile Data Management?
Agile Data Management and Agile Data Governance combine the philosophy of the Agile Manifesto with the goals of Data Governance: to ensure Data “quality, availability, integrity, security, and usability within an organization”. The Agile culture values early and continuous delivery to the customer, short delivery timescales, daily face to face communications among all team members, simple approaches to design that minimize work, and regular evaluations on how a team can become more effective.
In applying the Agile mentality to Data Governance we see a transformative, supportive, and collaborative Data Management team that self-manages short data projects that continuously move towards the purposes of Data Governance. Data Governance team members identify customer scenarios in data usage by active and daily participation. These form the basis in creating Data Models and Requirements. Data Governance focuses on providing the right data at the right time, based on user stories, and continuously tests and improves these Data Models.
Benefits of Agile Data Management
- Relevant Information
Information needs context to be valuable. For example, a firm providing reports to pharmaceutical companies needs to know what scenarios will drive the search for information. It may seem obvious, that pharmaceutical executives need a Big Data database system, including medical conditions and their approved drug treatments, to identify new markets. However, a significant amount of pharmaceutical business comes from off-label use, where doctors prescribe a drug, approved by the Food and Drug Administration (FDA), but for a different condition from ones approved by the FDA.
The Pharmaceutical business has interest in off label prescribing. IT would provide the most relevant information to pharmaceutical executives by conversing with people in the drug industry and finding use cases, perhaps around off label uses, identifying new markets. An American pharmaceutical has already taken the initiative in a two-year data-transformation program using Advanced Analytics, eventually towards gaining competitive advantage.
- Information Available Sooner
Engineers need to have up to the date research to make the best use of emerging technologies. These new technologies require updates to the Data Glossary and Dictionaries and accessibility by team members across the organization. Already, new engineering developments are expanding to include new concepts of autonomous systems, microsatellites, and organs-on-a-chip. As research unfolds rapidly, periodic importing the latest data in a comprehensive system may not be soon enough.
Agile Data Management lends itself to current awareness, notifying scientists about new discoveries, relevant to a project, immediately after the information is released. Through daily stand-ups, including the researchers and Database Administrators, understanding will emerge on how new scientific findings will create new information requirements and priorities for an organization. Data Lakes may result from these Agile Data Management processes, a promising tool to get information quickly to the end user. Agile Data Management addresses the engineers need for accessible current data and provides a solution towards developing effective Business Intelligence and Analytics by providing relevant data, a top IT issue.
- Better Responses to Information Changes
In the course of any project, information needs may change at any point. Agile Data Management provides the means to handle these changes. At the beginning of the Seated for Safety project, for the AAA Foundation, staff collected child passenger materials aimed towards the public and predicted educational gaps in disseminating the information to non-English speakers. As the project progressed researchers, librarians, and the Database Administrator met to communicate about new findings. The child passenger safety documentation did not address low income needs around fitting child passenger seats in older vehicles or people with low literacy. Researchers then needed to adjust study methods to evaluate these issues further.
This experience, of changing information needs, falls in line with that identified by IBM Agile Information Governance Process: “Data may be quasi- or ill- defined and subject to further exploration, hence critical data elements may change iteratively.” Adopting Agile Data Governance allows for information needs to evolve and change, allows for project members to jump in, use data and adapt to new discoveries, providing the right governance pattern to handle each situation.
Pitfalls of Agile Data Management
- Lack of Documentation
Agile Data Management can lend itself to a lack of documentation and a paltry Data Dictionary or Glossary if not completed with focused due diligence. Agile Data Governance focuses primarily on customer need. Upon providing the required information for a business problem, the next iteration, or time-period starts, with a focus on the new or updated customer need, with little time to create or record results on accessing information, on its integration or security. If a Data Dictionary or Glossary has been constructed, the context of the Metadata gained from the project may be lost, leading to gaps on what the information did in the first place. Lack of documentation resulting from inadequate Agile Data Governance application can have far reaching consequences, According to a Compuware Survey “seventy percent of CIO’s say that a lack of documentation will hinder effective knowledge transfer.” Agile Data Governance needs to consider how documentation will fit in each cycle, to allow for effective future use.
- Cumbersome to Implement in Larger Organizations
A health insurance organization uses the Agile methodology to implement surveys to individuals covered on the plan. These survey takers receive rewards based on their responses. Developers, software testers, business owners, and user design folks meet in daily stand-ups to sketch survey questions, responses and reports. However, the business owners work at a different location and time zone than IT. While user stories, documentation, emails, and instant messaging helps, gaps in exchanging information about business need and development efforts remain. Big businesses, which have offices spanning multiple locations, need a blueprint to implement effective Agile Data Management. Bradley de Souza makes this observation in the article Agility Comes with Maturity. Without a plan, information will remain siloed in departments in large organizations.
Agile Data Management: Best Practices
Tami Flowers suggests a methodology when using Agile to establish a Data Governance Organization Framework. Her recommendations include:
- Engaging data owners, customers, and stakeholders early on the process, and develop user stories
- Establish prioritizes and time lines around these priorities
- Throughout the implementation of a Data Governance Framework periodically look for ways to improve the processes
The US Government has come up with strategies on how best to practice Agile Data Governance through the report The Federal Big Data Research and Development Strategic Plan. This document cites successful Agile Data Management during the DeepWater Horizon oil spill event. During the event, National Oceanic and Atmospheric Administration’s (NOAA) CIO Joseph Klimavicz spearheaded the availability of an environmental, open source, ERMA Geospatial Platform.
He gathered in other government agencies and arranged for accessibility, early on, by the public and scientific sectors. The Coast Guard could monitor its clean-up effort and communicate these scenarios to other agencies and the public. Weekly, new features and data layers are added, a continuous process in meeting end-user needs and addressing new information requirements.
The Information Governance following the Deep Water Horizon oil spill demonstrates how and why data needs to be nimble. Agile methodology provides Data Governance tools towards achieving this requirement by business owners and stakeholders so operative Agile Data Management can move forward successfully.