Detecting new classes of anomalies when monitoring a high-dimensional data space is known as novelty detection. This is a special case of anomaly detection that can be extremely important in many real-world applications. Some examples for novelties are: a new class of customers that behave differently from others yet share common properties; a new manufacturing mode of operation in a semi-conductor line; and the introduction of new products to retail stores that generate a new consumer behavior e.g. due to cannibalism effects.
Although conventional anomaly detection and outlier detection methods are well used in many monitoring cases, these methods are less efficient for novelty detection for several reasons.
- First, novelties are unavailable within historical datasets and, hence, cannot be used to train or supervise the monitoring AI models.
- Second, novelties, unlike many conventional anomalies, do not behave like white noise that is spread over the entire space somewhat evenly and symmetrically.
- Third, the application of dimensionality-reduction techniques that limit the number of monitored features (before the novelties appear in the data) can lead to a lower dimension projection, where the novelties cannot be detected at all.
- Fourth, the high computational effort that is required to find the subspace where the novelties can be found, grows exponentially with the data dimension.
Now, consider for example the retail case, where the dimension size is defined by the number of products, reaching thousands or tens of thousands of SKUs – it is obvious that a conventional search is computationally prohibitive.
A new method, which is called Ensemble-Bayesian SPC (EB-SPC), is proposed in a new paper written with my former doctoral student Marcelo Bacher. The EB-SPC relies on the fact that novelties are often visible and correlated in subsets of attributes. It combines the ensemble-learning of classifiers such as Random Forest with Bayesian inference and a novel SPC paradigm. In contrast with regular monitoring approaches that often focus on a single class mode, the EB-SPC exploits the joint information of the trained classes and combines the posterior probabilities of various classifiers by using a mixture-of-experts approach. A performance evaluation on datasets from both public repositories and real-world datasets shows that the EB-SPC outperforms both conventional multivariate monitoring and ensemble-of-classifiers methods in novelty detection tasks, including the monitoring of multi-mode systems.
The paper addresses a specific example in the semi-conductor industry, in which the technology of producing integrated circuits is rapidly shifting in response to the high demand for new and faster products with ever-growing quality requirements. These shifts often result in novelties that can be captured by the proposed approach.
Alongside this specific use case, the EB-SPC can also be applied to retail monitoring at physical locations or in e-commerce sites. For example, it can be used to monitor features that are associated with products, such as sell-in, sell-out and inventory levels. I addressed several of these these examples, as well as other AI-based methods in my talk “Practical Applications of AI in Retail” at CB4’s NRF 2018 tech talk. See below for the full video featuring one of our clients, Joe Gauthier of Wesco.
