Billions of things with sensors surround people and their lives. These Internet of Things (IoT) interact with people, homes, factories, workplaces, cities, farms, and vehicles. Gartner predicts that by 2021, IoT technology will be in 95 percent of electronics for new product designs, from wearables to medical devices and beyond.

IoT promises useful information, allowing health issues to be detected sooner, fitness to be monitored, goods to be tracked better and more safely, and food produced more efficiently.

However, all these things create a lot of noise by sending large volumes and varieties of information at almost light speed. Managing all this IoT data means developing and executing architectures, policies, practices, and procedures that properly meet the full data lifecycle needs, which poses unique challenges. Traditional big data approaches and infrastructure need to be rethought and expanded.

Common Problems in IoT Data Management

Working with Internet of Things data requires a shorter time span than with data collected from humans. For example, survey data from people’s comments and actions tend to arrive in a matter of minutes, hours, or days, rather than seconds. Given this, managers may have had a little more flexibility in deciding which data to select, and less irrelevant data slipped through. In contrast, IoT is creating its own ecosystem, exacerbating three typical Data Management problems:

• Scalability and Agility: The sheer size of IoT data traffic and its immediacy makes this Data Management issue most pressing. Given that the number of IoT devices will increase with time, say from 40 to 400 devices, how can an IoT architecture accommodate this? How can IoT be connected, allowing for real-time processing and analysis by people and things, as IoT data has a short shelf life? Once IoT data gets somewhere, how can it be stored, ensuring enough space for new information? How can inputs and outputs flow through sensors, without becoming clogged? Should IoT data need access to non-sensor data (e.g., metadata about users and passwords), then how can the thing gain and understand such information?
• Security: Gartner’s survey shows that security is a significant challenge for organizations planning and implementing IoT solutions. It estimates that through 2022, half of all security budgets for IoT will go to fault remediation. Preventing unauthorized access has become forefront. Newsweek reported that nearly half of all U.S. firms using IoT have been hit with security breaches, and the costs can be staggering — over 20 million for large firms.

But this is only one part of the problem. Organizations need to be compliant with national rules and regulations on securing data. One major regulation, the General Data Protection Regulation (GDPR), enforced since May of 2018, potentially leverages substantial fines for non-compliance.

To understand the complexity, take the example of a refrigerator that notifies its owners that eggs and milk have expired. Posting that information onto the internet without the owner’s consent, even just to archive it, would breach regulations like the GDPR. But a refrigerator owner may want their grocery store to ensure eggs and milk are in stock. Functioning sensors need access to appropriate information.

To improve energy consumption, the fluidOps Information Workbench connects sensors to energy ontologies, provided by Norway’s StatOil, with a German provider of energy management systems analyzing IoT sensor data. The introduction of security barriers that prevent essential information flow between these entities would be counterproductive.

• Usefulness: Lewis Kaneshiro states that data has the most value when it arrives, and it steadily decreases as data sits in storage. IoT relies on fast data, on getting the insights now. Functions such as adaptive maintenance, security monitoring, predictive repair, and process optimization rely on real-time data.
• Safety: Consider the fatality from Uber’s autonomous vehicle in Arizona. Because it was evening, the car’s sensors may have failed to recognize the pedestrian in the darkness to use that information to slow down. If the SUV’s sensors did recognize the pedestrian, then it still did not use that information effectively.
• Filtering: How should sensor data be filtered and effectively? What types of data filters should be used for what types of sensors? What about incorrect information recorded by a sensor (e.g. recording a pedestrian as another car)? How can such false data be discarded? How can the IoT’s data be checked for quality?

Given the complexity and urgency of IoT problems, it is no wonder that the global IoT Data Management market size is expected to grow from $27.54 billion in 2017 to $66.44 billion by 2022.

IoT Data Management: Strategies and Solutions

Fortunately, IoT Data Management strategies exist from past technologies and methods. Christy Pettey from Gartner quotes Ted Friedman as saying, “Many of the same Data Management infrastructure tools and technologies applied to more-traditional use cases can be leveraged in some fashion to support IoT.” For example, consider:

• Edge Computing: In edge computing, data is processed near the data source or at the edge of the network, while in a typical cloud environment, data processing happens in a centralized data storage location. By processing and using some data locally, IoT saves storage space for data, processes information faster, and meets security challenges.

Should a refrigerator need to lower the temperature to an owner’s requirements, this can be done on the home network or even just by the device itself. Information about a thermostat malfunction or a needed fix may be stored locally and pushed out to the manufacturer or vendor’s cloud environment for further analysis. When developers have a patch to the issue, it can be packaged and sent through the cloud and opened locally by the device, which would handle most of the code updates and intensive processing, perhaps through a microservice.

In the meantime, since only the needed amount of data travels outside of the home to the vendor or manufacturer, unauthorized personal information is less likely to be shared.

• Data Governance: Data Governance mitigates security risks by defining access to information. Data Governance describes the authority and control over managing data assets. Previously, Data Governance described an IT centric service. In the IoT world, Data Governance becomes more essential to every user.

A typical house hold does not have an IT department, so Data Governance becomes the responsibility of the typical consumer. Consumer education on how to effectively govern individual data will become paramount.

Anyone and anything needing to make decisions about setting or using a device needs high quality information that can be used. This additional Data Governance imperative is key to Data Governance 2.0. With IoT, Data Governance will need to become a common household term.

• Metadata Management: For IoT data to be useful, metadata plays an essential role. Metadata describes “data in context,” said Donna Burbank. Good metadata cues a device on what information to use at what point and how to use it.

Metadata also provides a core for automated systems to do deep learning. In the case of automated vehicles, metadata provides trip context, making driving safer, which may contribute toward saving a pedestrian’s life. Thor Olavsrud at CIO paraphrases Emily Williams as saying that “extracting meaningful insights and increasing operational efficacy will require flexible, integrated tools that allow users to quickly ingest, prepare, analyze and govern data.” Metadata Management will be essential in this.

Gartner predicts that the Metadata Management solution will grow as enterprises continuously manage the data generated by sensor types, functions, locations, manufacturers, and serial numbers.

IOT Data Management in the Future

While edge computing, Data Governance, and Metadata Management will help firms deal with scalability and agility, security, and usability, this provides only a start.

Ted Friedman, Vice President and Distinguished Analyst at Gartner, explains that one-third of IoT solutions will be abandoned before deployment due to lack of Data Management and analytics capabilities adapted for IoT. He notes that organizations need to modernize in several key areas, including adopting new Data Management technologies and platforms, in addition to creating new Data Governance policies. For IoT to thrive, Data Management must include more modern infrastructures and the technologies to support them.

Image used under license from Shutterstock.com