Anomaly Detection Using Ensemble Classification and Evidence Theory

Fernando Arévalo, M. Tahasanul Ibrahim, Christian Alison M. Piolo, Andreas SchwungIEEE Access 2023 · DOI

TL;DR

This paper introduces ECET, a novel approach for anomaly detection in industrial processes using ensemble classification and evidence theory. By quantifying uncertainty and employing a strategic pool selection for classifiers, ECET effectively identifies anomalies in complex systems, outperforming traditional methods.

What Problem Does This Paper Address?

Industrial processes, especially those in manufacturing and energy, generate vast amounts of data. Detecting anomalies or faults in these processes is crucial for optimizing operations, preventing failures, and ensuring safety. While ensemble classification (EC) offers a robust way to combine multiple models for better performance, building an effective EC presents challenges: selecting the right "pool" of classifiers, designing a suitable architecture for combining their predictions, and, critically, quantifying the uncertainty in their predictions. This uncertainty is key to identifying deviations from normal operating conditions, especially when dealing with "out-of-distribution" data (i.e., conditions not seen during training). Existing methods often struggle to reliably detect anomalies when faced with novel or unexpected operational states.

Methods

The proposed ECET (Ensemble Classification and Evidence Theory) framework tackles these challenges through several key innovations:

  • Evidence Theory (ET) for Fusion: ET, specifically Dempster-Shafer Rule of Combination (DSRC) and Yager Rule of Combination (YRC), is employed to fuse the predictions of individual classifiers. ET is chosen for its ability to model and combine uncertain information from different sources.
  • Uncertainty Quantification (UQ): A novel UQ methodology is presented to measure the uncertainty of individual classifiers after training and, crucially, the uncertainty of the ensemble during inference. This UQ is a primary indicator for anomaly detection.
  • Strategic Pool Selection: Instead of brute-forcing combinations, ECET uses a multi-criteria strategy to select the best pool of classifiers. This includes considering:
  • Performance: Using the F1-score as a robust metric.
  • Expert Area: Identifying classifiers that excel at specific classes.
  • Diversity: Ensuring the selected classifiers offer varied perspectives.
  • Pre-cut: A parameter to refine diversity selection.
  • Anomaly Detection via Uncertainty Tracking: The core anomaly detection mechanism relies on monitoring the ensemble's uncertainty during inference. If the uncertainty exceeds predefined thresholds, an anomaly is flagged. This allows the system to identify unknown conditions.
  • Architecture: A clear architecture is defined for both binary and multi-class ECs, detailing the transformation of classifier outputs into evidence, the fusion process, and the final prediction.

Datasets

The primary dataset used for evaluation is the Tennessee Eastman (TE) dataset. This is a widely recognized benchmark for process control and fault diagnosis, simulating a chemical plant with 21 fault cases and a normal condition. The authors use various subsets of this dataset for training and testing, including:

  • Easy Datasets: Containing fault cases (0, 1, 2, 6, 12).
  • Hard Datasets: Containing fault cases (0, 3, 9, 15, 21).
  • Binary Classification Datasets: Typically using the normal condition (0) and one specific fault case (e.g., (0,1) or (0,3)).
  • Multiclass Classification Datasets: Using multiple fault cases (e.g., (0,1,2,6,12)).

The dataset is split into training, validation, and testing sets, with the testing set used to evaluate both classification performance and anomaly detection capabilities by injecting unseen fault cases as anomalies.

Key Insights and Ideas

  • Uncertainty as a Red Flag: The most significant insight is the direct use of ensemble uncertainty as a primary indicator for anomaly detection. High uncertainty during inference strongly suggests an unknown or anomalous condition, rather than just a poor prediction of a known class.
  • Strategic Pool Selection is Crucial: The paper demonstrates that a well-designed pool selection strategy significantly impacts the ensemble's performance and efficiency, reducing the computational burden of testing numerous combinations.
  • Evidence Theory Handles Uncertainty Well: ET's ability to model and combine uncertain information makes it a powerful tool for fusing predictions from diverse classifiers, especially when dealing with noisy or incomplete data.
  • Anomaly Detection for Novel Conditions: ECET moves beyond simply classifying known faults to actively identifying entirely new or unexpected operational states by monitoring uncertainty.

Key Results

  • Robust Classification: The proposed ECET approach achieved comparable or superior F1-scores to existing methods on the TE dataset, particularly for multiclass classification and anomaly detection tasks. For instance, certain ensemble configurations (e.g., H5-1, H6-1, H6-4) achieved average F1-scores of 0.97 in classification.
  • Effective Anomaly Detection: The anomaly detection capability was demonstrated to be effective, with ECs correctly identifying anomalous samples. For example, in anomaly detection experiments, ECs like H5-2 achieved F1-scores around 63-64% for hard fault cases, outperforming unsupervised methods like Top-K DCCA.
  • Reduced Misclassifications: The use of uncertainty tracking helped in correctly flagging unknown conditions as anomalies, rather than misclassifying them as known faults, which is a common issue with standard classifiers.
  • Efficiency: The strategic pool selection reduced the number of experiments needed, making the process more computationally feasible.

Limitations and Future Work

  • Static Classifier Behavior: A noted limitation is that the base classifiers are trained once and exhibit static behavior. This means they might struggle with significant concept drift if the underlying process dynamics change drastically over time.
  • Mutual Exclusion Assumption: The current framework assumes faults are mutually exclusive. In reality, multiple faults can occur simultaneously, which might require further adaptation.
  • Future Work: The authors suggest several avenues for future research:
  • Automatic Anomaly Detection Update: Developing a system that can automatically update the ensemble classifiers based on detected anomalies.
  • Advanced Pool Selection: Exploring more sophisticated strategies for pool selection.
  • Feature Reduction: Investigating different feature reduction techniques to improve performance.
  • Training with Diverse Data: Training classifiers with more diverse data to enhance their robustness against concept drift.

Summary updated

BibTeX
@article{arevalo2023anomaly,
  author = {Fernando Arévalo and M. Tahasanul Ibrahim and Christian Alison M. Piolo and Andreas Schwung},
  title = {{Anomaly Detection Using Ensemble Classification and Evidence Theory}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3280048},
}
Figure