What is Continuous Intelligence?

For the past 10 years, the most successful business applications of machine learning have been quite ethereal: online advertising (e.g. retargeting, website optimization), high-frequency trading, yield management...

Machine learning has often disapointed ventures in the gritty world of operations:

• Algorithmic performance was not that high compared to traditional linear models.
• Very few prototypes made it to production.

We have come to understand that for hardcore business challenges, machine learning works best when integrated into a larger continuous intelligence solution.

Here is Gartner's definition of continuous intelligence:

Continuous intelligence combines data and analytics with transactional business processes and other real-time interactions. It leverages augmented analytics, event stream processing, optimization, business rule management and machine learning.

Could that be the next big thing?

***

This page provides an introduction to the four main features of continuous intelligence, and links to related third-party resources.

Streaming data (or event streams, or time series) is the raw material of continuous intelligence: data that accumulates over time, one data point at a time.

High frequency is a possibility, but not an obligation. From an algorithmic and decision management perspective, daily sales data or weekly inventory data are the same as instant sensor data.

What matters is:

• The sequential nature of the data, as certified by reliable time stamps.
• The time dependence of the business challenge - meaning that the sequence of events conveys as much information as the events themselves.

Collecting, storing, preprocessing and modeling streaming data is very specific. The following resources clarify these specificities.

• An introduction to the IT infrastucture architecture requirements for continuous intelligence
• A good overview of two common arcitectures for streaming data (lambda and kappa)
• Keep it simple! Standard technologies will often do the job - the example of Twitter streams and MySQL
• Our introduction to time series modeling on Medium

When building continuous intelligence solutions, the single most important step is the definition of the target: what do we want to understand? which outcome do we want to affect? are there one or many data streams matching that definition?

But a key feature of continuous intelligence is that understanding the target will require processing other data streams for context. For example:

• Optimizing natural gaz procurement will require analyzing LNG price movements, but also understanding how oil prices, electricity prices, weather conditions, macroeconomic factors... affect LNG prices.
• The root-cause analysis of a manufacturing anomaly will require modeling that anomaly's behavior, but also measuring the impact of hundreds of potentially relevant parameters, on and around the production line.

This combination of target and contextual data has three notable consequences:

• It makes it hard to build continuous intelligence solutions without machine learning, since the more data sources, the less linear models will perform (the famed "curse of dimensionality").
• It means that continuous intelligence solutions will receive inputs from multiple internal and external sources, which creates specific IT challenges.
• Data gathering and processing are not free, so how do we identify the data that truly contributes to our continuous intelligence solutions's efficiency?

Resources on machine learning and data selection:

• McKinsey's executive guide to machine learning
• A technical but good discussion of feature selection techniques

Continuous intelligence places a unique burden on data analysis, for two reasons:

• The goal is to improve (and potentially automate) real-time decision making. This means that our analyses must update every time new data appears ("streaming analytics" or "streaming BI"), which in turn has significant repercussions on IT and algorithms.
• Because of the nature of time series, the true relevance test for time series analytics is prediction accuracy. Faithfully representing and analyzing streaming data necessarily implies the ability to predict upcoming values. In other words, for continuous intelligence, you can't be prescribe if you don't predict.

• A great introduction to streaming BI, including some historical context
• Useful reminder: the analytics pyramid

Continuous intelligence is decision-oriented: it is about helping managers select the right course of action when multiple factors and potentially conflicting goals are involved.

An important consequence is that continuous intelligence solutions require the combination of many types of models:

• Linear and machine learning models to make sense of the streaming data - internal and external.
• Business rules reflecting the company's policies (and often embodying its operational experience) in the corresponding area.
• Physical models describing the organization of the relevant assets.
• Human inputs! Capturing the expertise of operators.
• Optimization models formalizing the business trade-offs.

• An introduction to "decisioning" - the art of accelerating analytics to value
• A (free) Gartner webinar on decision management
• A simple example of the impact of combining business rules and predictive models
• An insightful HBR article on predictions, judgment, and machine intelligence

Datapred brings the power of continuous intelligence to industrial buyers of raw materials and energy - centralizing tens of internal and external data streams to provide buyers with a 360° view of their decision-making environment. Download the data sheet below to discover how.