Click to learn more about author Mathias Golombek.

We live and work in a world that has seen Big Data come to the forefront of nearly every sector of our lives. From medicine and mechanics to technology and retail, data gathering is big business, and now more than ever, it’s shaping the way we live.

Or is it?

There’s no denying that information is being collected, and with it, great advances are being made, but it might surprise you to hear that I am a bit ambivalent about the common understanding of Big Data. In an article a few years ago, Let’s replace the Vs of Big Data with a single D, I explained why I don’t like either the term nor its definition. What I can agree with is that Big Data in the sense of modern applications on top of data is one of the most exciting developments of the last decades. Unprecedented data volumes can be stored efficiently, and all kinds of great solutions have been built on top of that.

We read so many impressive stories about sensor data, the Internet of Things, Artificial Intelligence or Industry 4.0 revolutions. When you set the headlining stories aside, Big Data Analytics often boils down to smartly optimizing businesses, or even creating whole new business models.

We have seen a lot of buzz around Big Data and rising technologies like Hadoop’s scalable HDFS storage, NoSQL/NewSQL systems for data processing and streaming solutions like Spark or Kafka in the past few years. The industry is talking about it and many businesses have these technologies already in place while others are planning to implement these kinds solutions.

Still, it is important to keep things in perspective. The fact is that a large percentage of technical managers have plenty of stories about failed projects, huge costs or data lakes that became so huge that the data could not be leveraged properly. This is why so many decision makers grow cautious about Big Data and are reluctant to start new projects in that area. Here are three common fears about Big Data and my thoughts why you shouldn’t (read: mustn’t) worry too much about it.

Fear: Big Data Projects Often Fail.

This is true. According to studies, 50-60% of Big Data projects do fail, either by not creating the expected business value, busting the budget or missing the planned schedule significantly. This failure rate is even higher than for normal software development projects. And yes, Big Data projects are very challenging. You need to ensure the right skill set on your (internal or external) team. Your team has to pick the best technologies and overall architecture for the specific scenario. Further, only a good project management process will empower all stakeholders to communicate appropriately, understand dependencies and bottlenecks, and make sure that timelines and milestones won’t be missed.

Here’s the good news: In the phase of trial and error, immature software solutions and a lack of technological understanding seems to be over. Hyped technologies are meanwhile better positioned and not glorified anymore to solve everything at once. A whole industry of consulting companies provides large workforces of specialists. And the dust is settling how such data teams should be structured and positioned in the organization to enable empowerment and success. So, the fear of failure shouldn’t bother you too much. What would happen if your direct competitor is faster and more successful in collecting and leveraging Big Data? Isn’t that your biggest possible failure? Data analytics is already a competitive advantage and will become even more crucial for business success in the future. Even if your project would fail, your organization will still learn a lot from that experience and hopefully do it better the next time. And you can be assured that “next time” will come quickly because Big Data won’t vanish anymore.

Fear: Big Data is too Expensive

There are innumerable stories of projects where millions of dollars have been burnt for large hardware clusters, software licenses, and human resources without any positive impact on the company’s revenues. Whole departments have been ramped up for the sake of Big Data projects and failed afterward to create valuable insights. Despite the fact that many Big Data technologies are open source projects, you must not underestimate the actual costs. The challenge of such technologies is that you need a large number of developers or technical experts, rather than a few people who simply use certain products. Collecting, processing, storing and analyzing data is an extremely complicated task, and the Big Data technology has to be integrated deeply into your infrastructure. After having implemented such a system, new challenges will quickly arise. For instance, scalability if your data volumes explode or more users and applications are connected. That’s why your teams need to constantly invest in systems and constantly survey the market of technologies.

Still, measuring Big Data by its sheers costs would be a silly thing. As mentioned before, it is a crucial competitive advantage to leverage your data in the best way. Even if you are in a traditional industry, things can be revolutionized rapidly, as rising stars such as Airbnb or Uber have proven in pretty conservative markets by creating innovative services based on data.

You probably don’t want to be surprised by your competitors similarly, and that brings me directly to the third fear…

Fear: Having no Clear Business Objectives

There have been C-level executives that decided to run Big Data projects simply because everybody is doing it, rather than having a clear strategy. Sometimes it is indeed not easy to find these revolutionizing applications we read about in the newspapers.

Yet, many great inventions are discovered accidentally. If you don’t engage with Big Data at all, you can be sure that you’ll not invent anything. That’s why I recommend to start looking for data in your company, finding ways of creating new data across your business chain, finding ways to store, analyze and visualize your data. You’ll see how new questions will arise, how data can give you valuable insights of the history, your current situation, and the predictable future. And then you’ll enable yourself to find small improvements or maybe even revolutionizing new business ideas.

So, while there might be significant fears, challenges, and obstacles around Big Data, it would be a huge mistake to worry too much about them. You’ll find more expertise, support and technical solutions out there than ever before. And you’ll see that Big Data will create plenty of potential opportunities for you.