Abstract
Detecting rare events in highly imbalanced multi-class datasets is a critical challenge for Cyber-Physical Systems (CPS). Standard machine learning models often exhibit bias toward majority classes, failing to identify infrequent but high-stakes anomalies. This paper introduces a novel ensemble weighting scheme based on Mixed Integer Programming (MIP). By optimizing classifier weights on a granular per-class basis and integrating Elastic Net regularization directly into the optimization objective, the proposed method enhances generalization and sparsity. Experimental results across four benchmark datasets demonstrate a 4.53% average improvement in balanced accuracy over state-of-the-art techniques, while providing significant computational speedups suitable for edge deployment.
Introduction
In critical infrastructures such as power grids and water distribution networks, detecting "rare events"—including equipment faults and cyber-attacks—is paramount. However, the data generated by these systems is inherently imbalanced; normal operations provide millions of samples, while critical failures may offer only a few. Standard classifiers often become biased toward the majority "normal" class, leading to poor performance on minority classes.
While ensemble learning can mitigate these issues by combining multiple models, existing weighting schemes face several hurdles:
- Skewed Distributions: Difficulty in optimally balancing contributions when class distributions are extremely imbalanced.
- Resource Constraints: The need for efficient model selection from a large pool for deployment on "edge" devices.
- Overfitting: A tendency to overfit the validation sets used for weight calculation.
Methodology
The proposed methodology consists of three distinct phases centered on a Mixed Integer Programming (MIP) formulation.
1. Training and Validation
A diverse set of $n$ classifiers (e.g., SVM, Random Forest, Naive Bayes) is trained using stratified k-fold cross-validation. This process generates a mean validation accuracy matrix $V = [v_{ij}]_{n \times m}$, where $v_{ij}$ represents the accuracy of classifier $i$ on class $j$.
2. MIP-Based Weight Calculation
The core innovation lies in using MIP to solve for an optimal weight matrix $W = [w_{ij}]_{n \times m}$. Unlike traditional methods that assign a single weight to each classifier, this approach assigns weights to classifier-class pairs. This allows the ensemble to leverage "specialist" models that excel in specific minority classes.
The optimization objective incorporates Elastic Net Regularization, combining L1 (Lasso) and L2 (Ridge) penalties. This promotes sparsity—effectively ignoring weak models—and robustness, preventing any single model from dominating the ensemble. Additionally, binary decision variables are used to select a subset of the best $K$ classifiers from the available $n$, ensuring the final model is compact enough for edge devices.
3. Test and Evaluation
The optimized weights and selected $K$ classifiers are used for inference on the test set. This phase evaluates the ensemble's ability to generalize to unseen data.
Experimental Results
The approach was evaluated on four datasets with varying imbalance ratios ($\rho$): LeakDB ($\rho \approx 6060$), NSL-KDD ($\rho \approx 1223$), SG-MITM ($\rho \approx 5.9$), and CIC-IDS2017 ($\rho \approx 2.1$).
Accuracy and Generalization
The MIP-based method consistently outperformed six established weighting schemes, including Bayesian Model Averaging (BMA) and Differential Evolution (DE). It achieved an average balanced accuracy increase of 4.53%, with gains reaching up to 7.31% in highly imbalanced scenarios. Case studies on the NSL-KDD dataset confirmed that the MIP correctly identified and weighted "specialist" models (e.g., an SVM for specific minority classes) that would have been undervalued by global weighting schemes.
Computational Efficiency
For edge deployment, the speed of model selection is crucial. As the pool of available classifiers $n$ grows, the MIP approach demonstrates a significant speedup compared to exhaustive search methods.
As shown in the figure above, the MIP formulation allows for rapid identification of the optimal $K$ classifiers, making it viable for real-time updates in field-deployed CPS.
Conclusion
This paper presents a robust MIP-based framework for ensemble weighting in imbalanced multi-class environments. By leveraging per-class granularity and Elastic Net regularization, the method achieves superior accuracy and efficiency. Future work will focus on automating hyperparameter tuning for the Elastic Net components and exploring dynamic weighting schemes that adapt to input characteristics during inference.
Summary updated