Many BI managers, CEOs, and CIOs cannot afford to add more staff, so they are seeking technologies that can help their existing teams operate more accurately and efficiently. They “need to change the physics, as we call it. They can’t just add more people to the team,” said Amnon Drori in a recent DATAVERSITY® interview.

To have manual work done on premise using professional services combined with a variety of different systems performing a multitude of different processes to the data, including ETL, database, data warehouse, analysis and reporting tools, all from a multitude of different vendors. “At the end of the tunnel is this business user, who looks at a report after the data has gone through this journey, and hopes that it’s accurate,” he commented.

Drori says he is a “big, big fan of data.” In his 20 years working in tech, his method for decision-making has always been to collect data, learn the meaning of that data, draw conclusions, and then put the conclusions into action. Yet over the past 15 years, the amount of data as well as their different types, systems, and sources “have become so enormous that it has become very, very difficult to try to understand the data.” Amnon Drori, who is CEO and Founder of Octopai, recently discussed the challenges organizations face trying to find insights with the increasing size and complexity of modern data stores. 

Same Report, Different Results

Drori had an experience that has become a fairly common occurrence among his colleagues. He had brought a report to a meeting showing that he’d acquired 33 new clients in the previous quarter. The CFO arrived at the same meeting with a supposedly identical report saying Drori had only acquired 22 clients — a $2.5 million discrepancy. The CEO demanded that they find the correct number, yet it took more than two weeks for the BI team to discover that a business process had impacted one report but not the other.

This scenario has been playing out in companies as the massive amount of data becomes more than existing reporting systems can handle. Small discrepancies become magnified exponentially and providing business users with accurate information in a timely fashion is becoming increasingly difficult, he said.

Growing Concern Over Metadata

Metadata Management has become an area of even greater risk. “Metadata, it turns out, can win or lose lawsuits, send politicians to jail, and even decide medical malpractice cases,” said Drori in Bad Metadata Can Get You in Legal Hot Water. What might be inadvertent inconsistencies in different databases could lead to serious accusations of fraud or data tampering in a courtroom situation. Yet finding discrepancies in data spread across multiple systems, each with different internal rules, is extremely difficult, he said. Some companies are attempting to get a handle on their metadata by hiring more staff, while others are looking for an automated solution.

Changing Needs

Businesses are expecting more out of their IT investments than ever — it’s no longer enough to just provide numbers on a report. Drori’s clients tell him that they invest in automation because they want to:

• Understand how data lands in the reports they use
• Manage or leverage automation as they move from on-prem to cloud
• Rely on automation to update business processes
• Know in advance the possible impact of changes upstream to prevent future problems
• Consolidate different systems

Up until about five years ago, he remarked, people were fascinated with automation, and reactions to new developments were often met with “I didn’t know automation could do all of that!” Since then, automation has evolved from an astonishing new concept to a demand. Companies now assume that automation is capable of providing a solution, even if they haven’t yet seen it.

The Push to the Cloud

Amazon, Google, and Microsoft are working to convince organizations to move their ever-expanding on-premise data to the cloud, Drori said. If this trend continues, tools to help organizations manage and trust their data as it’s being generated and managed in the cloud will be critical. More and more organizations want to find meaning from their data and use it to make better decisions, yet gaining those insights usually requires large investments of resources.

Complexity Creates Pain Points

Over time, companies have added on multiple solutions to legacy systems in an attempt to manage rapid increase in the size and types of data stores.

“With the combination of the growing number of use cases in an expanding, ever-changing data environment, the BI team just can’t handle this anymore,” he said. “Even if staffing costs were not an issue, hiring more staff is not the only answer. The job is too big no matter how many people you hire.”

Challenges to Data Governance and Data Quality

David Loshin, President of Knowledge Integrity Inc., said that companies investing in Data Governance policies may find themselves struggling to implement them without adequate tools to trace data lineage. Data stewards are often given critical Data Quality Management responsibilities without being given the proper training or technology.

Loshin writes in How Data Lineage Tools Boost Data Governance Policies that without a way to determine where data errors are introduced into the environment, data stewards and quality analysts will find it difficult to identify and fix them.

“That has consequences: If data flaws continue to propagate in systems, the organization may be plagued by inconsistent or inaccurate analytics and reporting that lead to bad decision-making in business operations.”

Octopai’s Automated BI Intelligence Platform

The idea for Octopai arose out of a desire to “dramatically impact the way the market works.” As a result, many of the features offered are a direct contrast to solutions in the current landscape. Octopai relies on a very deep technology to thoroughly analyze different types of metadata, providing a cloud-based SaaS out-of-the-box product covering all vendors on the BI spectrum. “Our pivot is looking at the information about the data. We see it as a goldmine of insights,” said Drori. 

The platform comprises automated data lineage, data discovery and business glossary and enables BI & Analytics teams to gain complete visibility and control of their data, to get the full story behind their data so that they can deliver faster and more accurately to the business. Companies can create a catalog or business glossary containing all the metadata from their reporting system, and cross-platform access provides a 360-view of metadata across the entire BI landscape.

“There’s a deep type of lineage analysis. Users can make discoveries and find answers using a tool that behaves like a search engine.”

No special training or organizational process changes are required to implement it, he said. “You don’t have to invest a lot of time and capital to get it up and running. You can get Octopai up and running by investing 30 minutes of your time to extract the information about the data.” Within 24 to 48 hours, his users are able to get access to all the data elements that exist within their BI landscape. Searching for reports or references provides a view of the complete data flow.

Crowdsourcing Wisdom

There is value in the industry-level information that Octopai analyzes for their clients, and customers are interested in sharing best practices within their peer group, he said.

“Before you do anything in an organization, you want to have a reference point, and your reference point is going to be what others are doing. We can help you with that understanding.”

A financial sector client recently contacted Drori about Data Quality issues affecting their reports because they had more than 70 incidents from business users due to mismatched data. They wanted to know if others in their industry were dealing with similar issues, and how much time and money they should allocate toward Data Quality. Because Octopai works across multiple market sectors, Drori was able to provide information about how the best-in-class for the financial sector in that region prioritizes Data Quality issues, without discussing specific clients. “We have the information to help organizations funnel their investments into areas that can make them better.”

Expanding Coverage

Drori said that they to add support for more third-party tools. Recent expansions include Tabular on Azure Analysis Services, IBM’s Netezza, Vertica, and expanded support for OLAP cubes. More options are in development, he said, because, “If there’s one thing I like, it’s to leverage the latest technologies to help people to do their jobs better.”

Image used under license from Shutterstock.com