Every generation of data infrastructure technology has promised more speed and agility, or better standardization, centralization, and control. This “or” proposition has fueled the ongoing battle between business leaders who require timely insights from dynamic data sources to drive better outcomes, versus IT leaders who need solutions that provide high leverage, scale, and governance. Essentially, what is needed are data solutions that turn this “or” proposition into an “and” proposition.
Enter the modern data stack, which empowers business leaders with the autonomy to use data to drive revenue growth, improve productivity, and create competitive differentiation and enables IT leaders with the control to provide cost-effective, reliable, and easy-to-maintain data solutions.
Today, best-in-class organizations maximize their use of data to improve business performance, customer experience, and product adoption while providing the standardization, centralization, and governance needed to ensure high levels of security, reliability, and usage. These organizations are doing this with powerful, fully managed, and automated SaaS data infrastructure platforms.
Most mid-sized companies typically interface with 30 to 40 different apps or APIs feeding data into their data warehouse. Large enterprises can even have hundreds of APIs and web apps in their infrastructure. The latest trend is for companies to move to cloud data warehouses such as Snowflake, Google Big Query, or Amazon Redshift, as well as data lakes like S3. But the reality is a third of companies are still using legacy on-premises data warehouses, and are still relying on legacy and custom-created ETL tools and procedures to consolidate and transform their source data into actionable business insights.
As data gets transformed, it loses granularity and becomes less flexible and less useful for new data products or projects. It also becomes harder to move that data to a cloud SaaS data store – it’s like you’re putting slow “stone tires” on your screaming fast “Tesla” database. The ETL process makes the data pipelines within schema more fragile, and daily API changes make it harder to keep your data timely and relevant. You can’t rely on your pipeline if it keeps moving every day!
The team at Oldcastle Infrastructure ran into this problem when providers changed their APIs, breaking data pipelines and preventing Oldcastle from getting a cohesive view of their operations. They had visibility into their on-prem SQL Server database and NetSuite ERP individually, but they didn’t have a single view across both ERPs to view transactional, manufacturing, or production data collectively. About 90% of their business data was in one on-premise ERP in SQL Server, while 10% lived in NetSuite in the cloud.
“NetSuite is constantly changing its API, making conventional pipeline strategies difficult to maintain, especially when it has been, and continues to be, customized,” said Nick Heigerick, IT Manager of BI for Oldcastle. He initially thought he would have to pay someone ongoing fees to monitor the data flow in the background, but he was able to keep up with the NetSuite API changes, grab all of the data, and automatically centralize it into Snowflake by using a modern data stack.
The Modern Data Stack
The latest approaches to Data Management can achieve both the granularity that will enable long-term value with the real-time access that will satisfy business teams. We know that companies that manage their data best will win, and the more powerful, detailed, and accessible that data is, the more data can do for the business.
Large enterprises face the challenge of out-of-date data due to batch processing and business groups that don’t understand why they can’t get faster results from their Data Science team. They also face struggles getting data out of large systems such as SAP’s offerings. At the same time, small to medium-sized businesses (SMBs) need to look at how they capture and store their data for long-term growth, and make the right decisions early on, so they don’t have to rip and replace systems down the road. Small businesses can take advantage of subscription pricing from Data Management companies so they can limit their expenditures while they grow. Larger companies need to make sure they can rapidly replicate and manage growing volumes of data.
What’s the Solution? Data Convergence
Even with business applications changing quickly with new cloud technologies, there’s still a way to combine reliability and agility in Data Management. Companies need to ask how to make the right choices when building a modern data stack that will support their growing data store and expanding business. They need to combine speed and security, and they need to move past ETL and save their data for transforming later in the process, facilitating new purposes for that data down the road.
As data becomes more valuable to organizations large and small, companies will be best served if they can update their processes to run more efficiently, while also preserving as many important details as possible. This doesn’t have to be an either/or type solution – businesses can have both. The best path forward for organizations of all sizes and complexity levels is to provide capabilities that deliver a convergence of speed and centralization, agility and standardization, and autonomy and governance. Effective data capabilities are now table stakes for companies across all sectors – and winning the battle of agility and centralization will deliver a durable competitive advantage over time.