Big Data describes massive amounts of data, both unstructured and structured, that is collected by organizations on a daily basis. This Big Data can then be filtered, and turned into Smart Data before being analyzed for insights, in turn, leading to more efficient decision-making. Smart Data can be described as Big Data that has been cleansed, filtered, and prepared for context.

There are two primary kinds of Smart Data often discussed by experts in the industry. One form is information picked up by a sensor, and then sent to a nearby collection point, and acted upon, before being sent to an Analytics platform. This data is sourced from Smart Sensors, especially within the Industrial Internet of Things (IIoT) systems. The other kind of Smart Data is Big Data that has been processed and is waiting to be turned into actionable information. For purposes of this article, data going to, and from, a Smart Sensor is “sensor data.” The term, Smart Data, will refer to Big Data that has been screened for useful information.

Smart Data is a new tool for business. Big Data gets turned into Smart Data when it is collected and optimized, using the specific needs of the industry and the individual organization. The following areas are some of the use cases for Smart Data:

• Customer Journey Analytics weaves hundreds of customer internet interactions together from across multiple channels. It combines thousands of events to create a journey for a business’ customers. It is a data-driven approach used to discover, analyze, and influence the customers’ journey. (However, when the information is “wrong,” it is both irritating and insulting, and may cause the loss of a customer.)
• The Customer Experience analysis (or Voice of the Customer Analytics) uses tools and techniques to gather the customer’s attitudes, opinions, and emotions. Voice of the Customer Analytics emphasizes the mental state of customers. Other forms of Analytics normally focus on a customer’s actions and behavior, rather than their thoughts. Marketing organizations will often use this kind of analysis to manage reputations, manage products, and provide competitive Business Intelligence. Techniques for collecting this kind of information include short surveys and comprehensive software platforms.

Smart Data and the Five Vs

Big Data is commonly described as using the five Vs: value, variety, volume, velocity, veracity. A reduction in “volume” takes place with Smart Data. Only useful information for solving the problem is presented. Variety may, or may not, be reduced, depending on the screening process used in filtering the data. Value, velocity, and veracity (accuracy) should all increase with the decrease in volume.

Machine Learning and Smart Data

Machine Learning is often a training process for Artificial Intelligence platforms, but can also be used as a recognition and decision-making program. As the use and popularity of Smart Data has increased, it has also been used with Machine Learning algorithms designed to seek out Business Intelligence and insights. Machine Learning allows organizations to filter Data Lakes and Data Warehouses, creating Smart Data in the process.

Traditionally, organizations seeking Business Intelligence from Big Data have used Data Scientists, who spend time searching for insights and patterns within an enterprise’s datasets. Machine Learning algorithms using “Unsupervised Learning,” and combined with Big Data, have made it possible to perform Data Analytics more quickly, and without the Data Scientist. Machine Learning algorithms dramatically increase the accuracy, speed, and intelligence of screening Big Data, and with feedback, can continue to learn and refine the “filtering process.”

Artificial Intelligence and Smart Data

During the screening and filtering process of creating Smart Data, decisions are made as to which data should be blocked, and which should be presented. Machine Learning and Artificial Intelligence (AI)  use specific criteria during this process. AI is an ongoing attempt to create intelligence within machines, allowing them to work and respond like humans. Artificial Intelligence has provided flexibility and can address unique goals. For example, financial services firms can use AI-driven Smart Data for customer analysis, fraud detection, market analysis, and compliance.

Smart Data and Big Data Use Cases

Smart Data and Beer

An organization called AB InBev (known as the world’s largest beer brewery group) has been acquiring companies to gain insights from Smart Data. They want to know everything about the behavior and habits of beer drinkers. It recently acquired a brewery group named Weissbeerger, a company which installs measuring equipment in taverns and bars. Their slogan, “turning drinks into data” expresses their interest in how much beer gets sold where, when, and why, and which flavors are popular at a particular bar.

Spokesperson Peter Dercon stated,

“Thanks to Weissbeerger, we can better support our hospitality industry partners, and we can tap into the evolving consumer needs even better. Data is power and the beer manufacturer fully understands that, especially in a saturated market where consumers are looking for new alternatives, like craft beers.”

The huge amount of data they are collecting supports their in-house market research, now considered essential in competing with the saturated Western markets. The “craft beer,” and artisan brewers, are the new competitors. The use of Smart Data allows AB InBev to innovate more quickly, and to spot trends quickly, such as RateBeer.

Smart Data and Healthcare

The goal of delivering healthcare providers Smart Data is to help them work smarter, rather than harder. Currently, providers are finding the use of Big Data to be confusing and overwhelming. They are having to deal with an endless stream of priority tasks used to improve Data Quality. They are collecting data, but not using it. Andy Slavitt, the CMS Acting Administrator, stated:

“Physicians are baffled by what feels like the Physician Data Paradox. They are overloaded on data entry and yet rampantly under-informed. And physicians don’t understand why their computer at work doesn’t allow them to track what happens when they refer a patient to a specialist when their computer at home connects them everywhere.”

Forbes has called data siloing “healthcare’s secret shame”, while Health Data Management has reported silos are holding back  patient outcomes and research breakthroughs. The reasons for health care siloing can be varied and complicated — patient privacy, platform incompatibility, and the costs to generate data. Siloing comes with very real consequences. For example, cancer centers have most of the up-to-date cancer data, but “do not share their data.” Siloing is unshared information, and as a consequence, the majority of cancer patients will not benefit from cutting-edge cancer research.

According to Shane Pilcher, vice president of Stoltenberg Consulting, the answer to this problem is Smart Data. Pilcher added:

“Only a very small percentage of healthcare organizations today seem to be leading the way in healthcare data analytics, while the vast majority are very early in the business intelligence/analytics process, or haven’t even started. As a result, (healthcare) organizations seem to see Big Data as something that’s off in the very distant future; for most of them, anything outside of five years is almost nonexistent.”

Clearly, there is a need for both a paradigm shift, and Smart Data technology, within the healthcare industry.

Collecting Smart Data

Organizations with less of an understanding of Big Data often collect everything, and then store it in a Data Warehouse, Data Lake, or often what amounts to a Data Swamp. They are collecting Big Data with the intention of using it “when they finally make the decision to use it.” While these organizations might believe they are collecting years of historical data, in reality, the data may lack quality, or quantity, or may even be in the wrong format.

Their resources would be better used collecting data that is relevant to their business. An organization can be smart about the data it collects and stores in its Data Lake. Data takes time and money to store, organize, and govern. Collecting Smart Data, rather than “all” data can be an efficient strategy for small and midsize organizations. Focusing on collecting Smart Data allows a business to use cost-effective solutions in processing it. Gathering only the important data can simplify the use of Self-Service BI tools, keeping staff from getting lost in masses of irrelevant information.

Collecting Smart Data is not only about eliminating excess data. Smart Data may come from a variety of different sources, and an agile organization can merge these resources to develop a highly  focused Business Intelligence model.

Photo Credit: Khakimullin Aleksandr/Shutterstock.com