Insight by Exception // Part 1
October 5, 2018
Author: Avora Team
Insight is a powerful concept, suggesting the ability to comprehend and understand a complex situation or problem. In a business context, insight is derived from data and informs the decision making process. It’s the basis of Business Intelligence (BI) but the depth, relevance and timely delivery of that insight can mean the difference between valuable intelligence and a missed opportunity.
This is the 1st of a 2 part blog series on Insight by Exception.
The major barrier to insight is timeliness. Unless the data is analysed and presented at the right time allowing action to be taken to remediate or exploit the information, that insight loses its potency. Processing data is prime cause of delay affecting data accuracy as the resulting lag may even skew results. Because of this it makes sense to minimise processing time by working with data in its raw state.
This is a radically different approach. Most platforms transform the raw data and convert it in such a way that it works with the software but cannot ever revert to its original state. This is disadvantageous because it prevents the data teams from being able to access and query data on the fly without impacting processing. Data scientists and analysts need to be able to dive into the code from time to time but can’t if that raw data has been ‘cleansed’ and is now under the auspices of the software vendor.
Raw data ingestion, on the other hand, has several advantages. Using an ETL (Extract, Transform, Load) process radically reduces the data preparation process, making it faster and freeing up resource so that data analysts are no longer spending time collating data and can focus on data interrogation. The ETL process removes dependency on any third parties such as data warehouses or the software vendor. Plus it makes the process data agnostic so that data can be pulled from any data source simply by using the appropriate connector.
Removing the data analyst from the data aggregation process does of course require automation. But why stop at the automation of data collation? Automation can also help improve speed to insight by generating reports in exportable formats such as Powerpoint, CSV, HTML, PDF etc. allowing these to be readily shared. This removes the possibility of data languishing on a desktop until the user gets round to exporting it and shortens the time taken to make data actionable intelligence.
Insight by exception
When the user reviews a report they’re looking for trends and how things have changed over time and they will then draw their own conclusions. This process too can benefit from automation. If we understand the business metrics it’s possible to map those out as historic trends. Applying Machine Learning algorithms we can determine tolerance levels within these metrics.
Take Sales, for example. Plotting revenue generation for the company will give a decision envelope with highs and lows that provide automatic tolerance levels. Those can be adjusted according to the sensitivity of the business but essentially you’ve got two thresholds that act as the tolerance parameters. That’s the basis for an alerts system that has the ability to flag incidents that occur outside those acceptable parameters – or what we call ‘exceptions’ – and to do so in real-time.
Why is this more productive than simply regular data report reviews? Because instead of looking at reports every day/week/month for events that exceed these thresholds, the user can carry on with business as usual (BAU) knowing that exceptions will be flagged as and when they occur. This enables the user to take immediate action, making the business both more effective and responsive. It’s this approach that forms the basis of AVORA’s Smart Alerts feature.
As BI matures it’s going to be this speed from data to outcome that matters. Even if you have an hourly or fifteen minute data feed a real-time alert capability will see the window between insight and decision diminish. Realtime reporting effectively means the business is constantly being monitored and is notified the moment a metric is impacted. That could mean you receive more alerts than you anticipated but then it’s just a case of simply adjusting the metrics you are measuring, and the system will automatically build out the charts for you.
For our next blog in the series “Insight by Exception” – we will focus on Root Cause analysis.