Abstract
A new semi-supervised ensemble algorithm called XGBOD (Extreme Gradient Boosting Outlier Detection) is proposed, described and demonstrated for the enhanced detection of outliers from normal observations in various practical datasets. The proposed framework combines the strengths of both supervised and unsupervised machine learning methods by creating a hybrid approach that exploits each of their individual performance capabilities in outlier detection. XGBOD uses multiple unsupervised outlier mining algorithms to extract useful representations from the underlying data that augment the predictive capabilities of an embedded supervised classifier on an improved feature space. The novel approach is shown to provide superior performance in comparison to competing individual detectors, the full ensemble and two existing representation learning based algorithms across seven outlier datasets.
Introduction
Outlier detection is a critical "needle in a haystack" problem. In most real-world scenarios, outliers are extremely rare, leading to massive class imbalance that often cripples traditional supervised classifiers. Furthermore, ground-truth labels are often expensive or impossible to obtain, making purely supervised learning impractical for many domains.
Conversely, unsupervised methods—such as k-Nearest Neighbors or Isolation Forests—are adept at identifying anomalies based on inherent data structures like local density or global correlation. However, these methods cannot leverage known labels to refine their predictions. XGBOD addresses this gap by proposing a hybrid framework that utilizes unsupervised methods to engineer superior features for a supervised ensemble, effectively treating unsupervised outlier scores as a form of representation learning.
Methodology
The XGBOD framework operates in three distinct phases designed to transform raw data into a high-performance predictive space:
- Unsupervised Feature Engineering: The original dataset is processed through a variety of unsupervised outlier detection functions (e.g., LOF, kNN, One-Class SVM). Each function produces a "Transformed Outlier Score" (TOS). These scores represent the data from multiple perspectives, including density-based and distance-based views.
- TOS Selection: To mitigate noise and reduce computational overhead, the framework prunes the generated TOS. The authors propose three strategies: Random Selection, Accurate Selection (choosing the highest ROC scores), and Balance Selection. Balance Selection is the most sophisticated, employing a greedy algorithm and Pearson correlation to ensure the selected features are both accurate and diverse.
- Supervised Prediction: The selected TOS are concatenated with the original features. This augmented feature space is used to train an XGBoost classifier. XGBoost is selected for its robust handling of imbalanced data and built-in regularization, which prevents overfitting on the newly engineered features.
Experimental Setup
The authors evaluated XGBOD on seven real-world datasets from the ODDS (Outlier Detection DataSets) library, covering a range of dimensions and outlier ratios:
- Arrhythmia (274 features, 15% outliers)
- Letter (32 features, 6.25% outliers)
- Cardio (21 features, 9.6% outliers)
- Speech (600 features, 1.65% outliers)
- Satellite (36 features, 31.64% outliers)
- Mnist (100 features, 9.21% outliers)
- Mammography (6 features, 2.32% outliers)
Key Insights and Results
Representation Learning Advantage
The core innovation of XGBOD is viewing unsupervised outlier scores as a rich, low-dimensional representation of the data's anomalousness. This provides a significant advantage in high-dimensional datasets (like Speech and Arrhythmia), where raw features are often too noisy for a classifier. Unsupervised scores distill this complexity into actionable signals.
Diversity and Visualization
The "Balance Selection" method highlights that ensemble performance relies on variety. Selecting diverse types of detectors provides a more robust view of what constitutes an outlier. Using t-SNE visualization, the authors demonstrate that adding TOS significantly improves the separation between normal points and outliers compared to using original features or TOS in isolation.
Performance Metrics
- Superior Accuracy: The full XGBOD ensemble outperformed all baselines—including individual unsupervised detectors and previous semi-supervised methods like BORE—on 6 out of 7 datasets in both ROC and Precision@N metrics.
- Significant Gains: On the Speech dataset, XGBOD improved Precision@N by over 23% compared to standard supervised XGBoost.
- Efficiency: By replacing the costly "EasyEnsemble" method used in prior work with a single tuned XGBoost model, the authors achieved more stable results with faster execution times on large datasets like Mnist and Satellite.
Conclusion and Future Work
While XGBOD shows state-of-the-art performance, it faces computational costs in Phase 1 when generating scores for every data point, particularly for high-complexity algorithms. Future research directions include exploring automated feature selection on the original data prior to TOS generation and investigating dimensionality reduction techniques, such as PCA, to combine unsupervised scores more efficiently than simple selection.
Summary updated