A 20-year-old article from MIT Technology Review tells us that good software “is usable, reliable, defect-free, cost-effective, and maintainable. And software now is none of those things.”
Today, most businesses would beg to differ. From payments to CRM to analytics and people operations, software runs everything. Businesses rely on data to drive revenue and create better customer experiences – and software helps them do it. Does data itself need to undergo a similar revolution to software, as it becomes ubiquitous across all sectors?
The seven principles of data reliability engineering (DRE) tell us how to improve data quality, keep it moving in a timely fashion, and feed machine learning products with healthy inputs for accurate analysis. This work – done by data engineers, data scientists, and analytics engineers and borrowing concepts from SRE and DevOps – helps guide data teams toward reliable data pipelines. It is the backbone of the data reliability revolution currently taking place.
Why Now?
In a broad sense, data quality has been a topic of interest for decades, but in recent years it has stepped into the spotlight in a major way. Data is used in higher and higher impact applications, from support chatbots to inventory management to financial planning and more. If these applications use data correctly, they can help organizations reduce risk and see massive ROIs.
Additionally, the virtualization of everything has meant that humans are now regularly taken out of the workstream. With streaming data, machine learning models, and self-service dashboards, the number of humans required in various workflows has decreased. That means that data scientists and analysts aren’t always available at every step of the data pipeline to spot-check data.
Lastly, there simply aren’t enough data engineers to go around. The demand for talent is skyrocketing, and the supply of people who are capable of building and scaling complex data systems can’t keep up. These teams need to be efficient, and so automation and tools that prevent data problems are growing more attractive.
Is It Data Reliability Engineering or DataOps?
Data reliability engineering sits under DataOps, but it’s only a part of the broader set of operational challenges that data platform owners face. DataOps challenges include issues like data discovery, cost tracking, access controls, and more. DataOps teams are often responsible for data reliability engineering issues like uptime and reliability, but they also often manage broader issues like developer velocity and security concerns.
What Is the Future of Data Reliability Engineering?
Data reliability engineering is a new concept. Many companies are helping to define the tools and practices that will bring data reliability engineering to the maturity levels of practices like SRE and DevOps. Reliability engineers are increasingly in demand, as the need for insights and reliability scales. As data and tech stacks grow in volume and complexity, it’s time to turn to the principles of data reliability engineering to effectively solve data challenges.