“When we’re talking about streaming analytics, we’re talking about flipping our traditional paradigm a little bit and thinking about how we bring analytics to our data, and not necessarily data to our analytics,” said Kimberly Nevala, the Director of Business Strategies for SAS Best Practices, while discussing streaming analytics during the DATAVERSITY® Enterprise Analytics Online Conference.

“Streaming analytics, as the word ‘stream’ implies, means that we’re bringing analytics to what we call ‘event streams,’” Nevala continued. Simply put, event streams are low-latency, high-throughput data-flows that generate insights by applying analytics to data while it is “in stream.” Traditionally, data is captured and stored before being analyzed, and insights that arise from that analysis are then pushed out. In streaming analytics, models or algorithms are applied to analyze the incoming data as it occurs, before the data is stored. This process provides the ability to “Interrogate the information and determine whether data has meaning and what the value is so that we know explicitly what data to store, and why, and when,” she said.

Why Streaming Analytics

Organizations are increasingly challenged to reduce the time from gaining insight to relevant action and to do so continuously in an agile way that is responsive to the environment. Nevala noted:

“That agility and responsiveness requires a different approach – to be able to not only ingest data, but to actually develop and apply insights very, very quickly . . . where we can continuously evaluate and respond to opportunities and avoid or address risk.”

As an increasing number of people interact with devices – and those devices become connected to each other – the ability to respond with complete awareness to an event or an instigating item in the moment is becoming increasingly important, Nevala said.

Providing a Real-Time Spectrum

Streaming analytics is a move from “reactive” real-time processing to “proactive” real-time processing. Traditionally, real-time decisions are triggered as a result of a pre-existing defined set of actions such as a purchase, a payment, or some type of system failure, and the real-time response comes in with some kind of pre-defined instruction. “We’re waiting for something to happen,” Nevala said. Streaming analytics and event-stream processing, conversely, is constantly analyzing data in motion before the data is stored. This includes actions like scoring, data manipulation, normalization, and cleansing. Most importantly, the process is focused on pattern detection, or changes in pattern.

“This is why streaming analytics is so important in things like fraud or cyber security, where, in fact, we don’t know exactly what that next action is or how someone’s going to attack next,” Nevala explained. “But we can actually start to look for differences in normal behavior and use those changes in pattern to tell us that something’s happening.”

The Impact of the Internet of Things

Nevala cited a McKinsey Digital estimate that the Internet of Things (IoT) will result in $11 trillion in revenue and business value by 2025, “so the promise and the potential here is huge.” In practice, however, the promise of IoT is not that easy to achieve. Simply generating massive amounts of data is not enough. Generating value from emerging trends like AI and machine learning requires the development of new insights from IoT data, she said, and from aligning and integrating IoT data with product data, customer data, and other sources that arrive in a variety of formats.

There is a huge cost associated with the transport and storage of IoT data, and because of the volume collected, it requires a strategy just to be able to store it. Because sensor data can describe almost anything, the variety goes beyond the simple classifications of “structured” and “unstructured” data. “So, storage may be seen as a commodity, but given the volume and velocity of the data, that cost becomes unsustainable,” Nevala said.

Streaming Analytics: Four Key Indicators

If more data alone won’t translate into value, then it becomes critical to discern what data is important. For example, knowing that a car is about to fail may be helpful, but having the information needed to fix the car before it fails provides true value.

Nevala outlined four situations where streaming analytics can play a role in creating value.

• Extreme Low Latency: When extreme low-latency response is of utmost importance, such as where machine failure could be catastrophic, with fraud detection, or with potential cyber-attacks.
• High Throughput Data: Streaming analytics is advantageous for real-time risk detection with big and high-throughput data.
• When Data Storage is Impractical: When the storage of massive amounts of data is not possible or optimal, streaming analytics can standardize incoming data, determine if it’s relevant, and if it isn’t, the event and the associated data can be discarded without taking up processing bandwidth.
• When Situational Awareness is Paramount: When an organization needs to take quick, appropriate action based on situational awareness.

The New Analytics Paradigm: Stream-Understand-Act

Traditionally, data is captured, stored, and put through some sort of ETL process in a data warehouse, or in some cases, into a lake. Reporting and analytic tools are applied against that data for insights, and that insight is fed back into the business process.

The new paradigm doesn’t entail getting rid of the original data-pipeline, Nevala said. Instead, it’s important to think differently about how to deploy analytics models. Applying high-end analytics on the event-stream itself doesn’t preclude the ability to look at off-line data to identify emerging trends. IoT data or real-time transactional data can be combined with traditional data sources to provide the context required to implement the analytics.

Use Case: Predictive Asset Maintenance

“Streaming analytics used to deploy advanced analytics techniques is really driving value in organizations,” Nevala noted, and predictive asset management was one of the first use cases for streaming analytics. Oil platforms require underwater surveillance, for instance, to avoid service interruptions and optimize production on the oil field. She estimated that one failed pump could cost $2 million a day, with a deferred revenue impact close to $20 million a day. When one valve or pump is failing, demand can be off-loaded automatically to other areas.

Use Case: Predictive Patient Management

Another use case is in health care, where clinically observable symptoms are sometimes not obvious until it is too late, Nevala said. Streaming analytics can detect relevant patterns developing in patients in real time, and that information can then be used to alert critical care teams.

Important here is the ability to avoid what’s called “alert-fatigue,” due to the overwhelming number of sensors and alerts throughout the day. “It’s very difficult to discern, even for medical professionals, which things are most important.” A patient’s vital statistics from those sensors can now be connected with incoming lab results, patient history and other data, which can then be used to trigger actions based upon detected patterns. The action triggered might be to send a message to the patient’s care team via email or SMS, allowing them to provide the right care at the right time. Streaming analytics are also being used to detect and treat life-threatening infections in babies in neonatal intensive care 24 hours before a clinical symptom would be observable.

Use Case: Contextual Customer Experiences

Advances in analytics are often driven from a desire to improve customer experience by providing more real-time context. Nevala talked about the impact streaming analytics had on a telecom provider. By combining customer records, web data, and geographical-based data in real-time, the company was able to provide marketing offers based on network usage, content programming, and recommendations. As a result, they have been able to increase offer acceptance tenfold and reduce customer churn. The ability to do this in real-time, in context, allows the company to be much timelier and more relevant by appealing to the customer at the time they’re using the service – “When they’re interested and potentially ripe for a new offer or an upgrade,” she said.

Use Case: The Connected Country

The Netherlands has one of the highest population densities in Europe, with 20 percent of the country actually situated below sea level and more than 50 percent of the country less than a meter above sea level. In such a flood-prone environment, transportation infrastructure management is critical. Consequently, close to 14,000 government employees are dedicated to the task of monitoring roads, bridges, tunnels, and waterways. To help manage and optimize transportation flow, they rely on real-time analytics from streaming data sent by bridge sensors. Employees use this data to safely change how lights work, manage which flood ways are on and off, and to change direction, she said. They plan to expand that analysis to include streaming data in real-time from thousands of sensors across their water-system infrastructure so that they can monitor and manage that critical infrastructure in real-time as well, Nevala said.

Putting it All Together

Nevala stressed that the insights gained from the technology should be used to inform business processes and support operational changes that will improve performance. She talked about being open to a change in mindset about how jobs, roles, and accountabilities may need to evolve, not just in data and analytics teams, but in how the business itself operates. Managing the necessary business changes proactively will ultimately shorten the time to value from investments in streaming analytics.

“The value here, like all other types of analytics, is in the action,” Nevala concluded. “It’s not [just] in developing the insights or detecting an event – it’s in understanding, ‘What do we need to do?’”  

Want to learn more about DATAVERSITY’s upcoming events? Check out our current lineup of online and face-to-face conferences here.

Here is the video of the Enterprise Analytics Online Presentation:

Image used under license from Shutterstock.com