Click here to learn more about author Sakthi Rangarajan.
In the recent months, I saw a few Data Warehouse efforts getting disrupted, stalled or stopped altogether. In one scenario, the effort was stopped in a few months and in another case, the project was paused after the Data Warehouse has been live for more than a year. In the third and an interesting scenario, the Enterprise Data Warehouse was getting abandoned and the funds are going for a smaller-scope solution based on an in-memory database. In all the three scenarios, the existing databases were on-premise commercial relational databases and they are all getting replaced not by cloud and/or open-source solutions but again by on-premise licensed databases. This made me think about what happened here and it all boiled down to the below reasons.
Lack of Business Ownership: Every single time I am at a client site, especially at the beginning stages of a Data Warehouse effort, I emphasize the importance of program ownership by the collective business team and not by IT. The role of IT is an enabler of the program, meaning, IT can make decisions on the technology, infrastructure, platform and the team model (internal expertise, offshore, remote, etc.). The business leadership is the one to define the need, requirements, capabilities, functionality, road-map and the strategic direction of the data warehousing program. The business SMEs should be involved right from the conception to the maintenance of the EDW. How often this happens? Not very often. This is a key reason why many EDW efforts fail. In all the above scenarios, business SMEs were involved very late in the game.
The case of ‘If you build they will come’ does not apply to a Data Warehouse. The user community is always wary of change and it is the natural human tendency too. If they are not part of the project and are brought in the later stages (training before go-live), they always tend to stick to their old ways of doing things and will expect the new solution to work as the old system did. When that does not happen, the next stage is complaints. When the noise will reach the senior business leaders, that will start ringing the death knell for the Data Warehouse.
Lesson learned here is that find the key business leaders, users and champions at the beginning of the EDW effort and engage them throughout the lifecycle.
Lack of EDW/BI Leadership: In many IT teams, there is not a senior level person who understands and leads the data warehousing and business intelligence program. At least, it is the case in the above three scenarios. This can truly hurt the investment made in the program. For example, there is a key difference between Business Intelligence and operational/transactional reporting. If IT leadership does not understand and does not govern the reporting requirements, it can lead designs in the Data Warehouse to support transactional reporting and efforts in the operational databases to support some level of BI reporting. This will negatively impact both the systems and will leave the users un-happy.
The development team will end up spending most of their time fire-fighting against the emergency of the day which leads to poor design/development practices. A strong EDW/BI leader should protect the effort against these distractions and help the team focus on strategic development.
Take away here is that IT needs a strong leader to lead the EDW and BI program. If there is not one, a leader should be hired to make sense of the investment and to deliver value from the EDW.
Lack of All-Around Best Practices: The core development team – architect, ETL developer and BI developer, needs to be aware of the EDW/BI best practices and should follow them. At least one person from each area – architecture, ETL and reporting, should be knowledgeable on the full EDW eco system. If this not the case, it negatively impacts the stability and work delivery rate of the system (the speed at which user requirements are satisfied). For example, if a business logic says a certain value need to be filtered from a metric calculation, that can be done in ETL, database views, semantic layer or in the reports. How do we decide where to do this logic? This is when the strong team is needed. If part of the team is not aware of or capable of doing best-practice based development, logic gets written in the wrong place leading to a slow and unstable system. This can also happen – well-thought out work happens in part of the system and doesn’t get carried all the way to the user, making the system appear less-capable.
Lesson learned here is when the development team is assembled, make sure the resources are well-aware and using the best practices and standards. A governance mechanism is also needed to facilitate design sessions to solve user requirements.
Falling for Technology Trap: I am strong believer that a technology does not solve a business problem but the correct application of the any suitable technology with engagement does. For example, a Data Warehouse built with Teradata can fail while one built with an open-source database solution like MySQL thrives. More often, the IT and business leadership fall for the pitch that having a specific database or reporting solution solves most of their reporting issues. I saw this happening in all the above scenarios. One firm decided to stop the Data Warehouse that was functioning on Sybase and decided to pursue a rapid mart solution provided by one of their ERP vendor. Here the scope of the solution got limited to that specific ERP data. There is also a lot of hype around Data Lakes (using Hadoop) and the new generation of NOSQL databases like Cassandra and MongoDB. These are great fit for specific Big Data analysis/modern applications (like IoT, social media, etc.) but they cannot replace an Enterprise Data Warehouse. They are still not meant for the generic users but for specialized data savvy analysts and scientists. I have seen more than one company where they tried to move their Data Warehouse to Hadoop and backed out of it in a few months.
Take away here is that don’t fall for the sales pitch or for the hype. Yes, IT should keep up with the latest advancements in the technology field, but be aware that technology alone does not solve a business problem.