What is Anomaly Detection?
July 20, 2018
Author: Ricky Thomas
The director of security for a large regional retailer looked over the employee behaviour analysis charts. Something wasn’t right – shrinkage had increased 2 percent in the last 24 weeks alone. Anomaly detection analytics indicated the same four employees in one city clocked in extra-early on the fourth Saturday of each month, the day of the monthly company-wide super sale.
Cross-checking shrinkage reports eventually revealed these employees conspired with outsiders to steal hundreds of thousands of dollars of TVs, electronics and computer equipment. By examining anomalies in employee data, the company was able to prevent further losses.
Anomaly Detection Definition
An anomaly is a pattern that deviates from the expected or normal result. Anomaly detection is locating patterns that do not behave as expected – it looks at clues and compares attributes to discover out-of-the-ordinary patterns. Many times, there are multiple anomalies in groups, not just single occurrences.
Examples of Anomalies
Take a look at some examples of real-world anomalies to get a better idea of how detection is used:
In ad-tech, anomalous pattern detection is combined with other methods, such as deep traffic analysis, to help keep illegal botnets from creating false ad impressions. Ad fraud is a major problem the online ad industry is fighting right now.
In tech, real-time analytics models scour sensor readouts to detect anomalous patterns than can help predict when different machine components will fail.
In retail, investigators detected anomalies when a criminal put fake barcodes on expensive toys in order to buy them for much less. He later sold the toys online at their real value and profited more than $600,000.
In hotels, managers monitor variance reports to detect suspicious activity and possible fraud in food and beverage services.
To cut down on recurring IT glitches in the travel industry, airlines are beginning to use anomaly detection and clustering to alert operators to problems before they happen.
Anomaly detection programs use algorithms to determine when a metric is acting in a way that is a departure from historical trends. The algorithms are on constant alert for irregular behaviour.
Different Types of Anomalies
There are several different types of anomalies:
Point – a data point is significantly different than others. Example: the location of an errant tree in an orchard.
Contextual – often found in time-series data, contextual anomalies are specific to a certain context. Example: buying much more sunscreen while on vacation is normal, but less so when back home.
Collective – groups of data points help find anomalies. An example is when a website suddenly experiences spikes in traffic.
Simple anomaly detection flags data points that fall outside regular patterns in a distribution – whether the median, mode, mean or quantiles.
Machine Learning Detection Methods
Machine learning anomaly detection methods include:
Density-based – when a data point falls outside a neighbourhood.
Clustering-based – when data points fall outside a similar group or cluster.
Support-vector – a training set is indicated to detect normal data clusters while testing sets look for abnormalities of the learned area.
Modern anomaly detection programs are complex, but you can do simple anomaly detection using a low-pass filter with a moving average.
Significant Cost Savings
Anomaly detection can help organisations realise significant cost savings. For example, a McKinsey Global study revealed that data-driven predictive maintenance will save manufacturer’s up to $630 billion in 2025. Similarly, the Deloitte University Press report “Industry 4.0 and Manufacturing Ecosystems” shows predictive maintenance and detecting anomalies related to machine failure could save companies like Caterpillar millions of dollars and reduce equipment downtime.
Changing the Game
Anomaly detection helps stakeholders throughout the organisation. Marketing managers can better predict future consumer trends, operations managers are able to predict equipment purchasing needs and maintenance failure, and C-suite leaders can better manage employee compensation, training and incentive packages. On every front, anomaly detection combined with predictive analytics changes the face of business.
AVORA provides anomaly detection with intelligent alerting to find the unknowns in your business. Have a look at our Smart Alerts feature, schedule a demo.