XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning

Yue Zhao, Maciej K. Hryniewicki

TL;DR

XGBOD (Extreme Gradient Boosting Outlier Detection) is a semi-supervised framework that enhances outlier detection by treating unsupervised outlier scores as a form of representation learning. By augmenting the original feature space with these scores and training an XGBoost classifier, the model achieves superior performance on highly imbalanced datasets compared to purely supervised or unsupervised approaches.

What Problem Does This Paper Address?

Outlier detection is a "needle in a haystack" problem. In most real-world scenarios, outliers are extremely rare, leading to massive class imbalance that cripples traditional supervised classifiers. Furthermore, ground-truth labels are often expensive or impossible to obtain, making purely supervised learning impractical.

On the other hand, unsupervised methods (like k-Nearest Neighbors or Isolation Forests) are excellent at identifying anomalies based on data structure—such as local density or global correlation—but they cannot leverage known labels to refine their predictions. The authors address this gap by proposing a hybrid framework that uses unsupervised methods to "engineer" better features for a supervised ensemble.

Methods

The XGBOD framework operates in three distinct phases:

  1. Unsupervised Feature Engineering: The original dataset is passed through a variety of unsupervised outlier detection functions (e.g., LOF, kNN, One-Class SVM). Each function produces a "Transformed Outlier Score" (TOS). These scores represent the data from different perspectives (density-based, distance-based, etc.).
  2. TOS Selection: To prevent noise and reduce computational overhead, the framework prunes the generated TOS. The authors propose three strategies: Random Selection, Accurate Selection (choosing the highest ROC scores), and Balance Selection. Balance Selection is the most sophisticated, using a greedy algorithm and Pearson correlation to ensure the selected features are both accurate and diverse.
  3. Supervised Prediction: The selected TOS are concatenated with the original features. This augmented feature space is used to train an XGBoost classifier. XGBoost was chosen for its inherent ability to handle imbalanced data and its built-in regularization, which prevents overfitting on the newly created features.
The three-phase XGBOD framework: Feature Engineering, Selection, and Prediction.

Datasets

The authors evaluated XGBOD on seven real-world datasets from the ODDS (Outlier Detection DataSets) library:

  • Arrhythmia (274 features, 15% outliers)
  • Letter (32 features, 6.25% outliers)
  • Cardio (21 features, 9.6% outliers)
  • Speech (600 features, 1.65% outliers)
  • Satellite (36 features, 31.64% outliers)
  • Mnist (100 features, 9.21% outliers)
  • Mammography (6 features, 2.32% outliers)

Key Insights and Ideas

  • Unsupervised Scores as Representations: The core "aha moment" is viewing unsupervised outlier scores not as final answers, but as a rich, low-dimensional representation of the data's anomalousness.
  • The Dimensionality Advantage: XGBOD provides the most significant boosts on high-dimensional datasets (like Speech and Arrhythmia). In these spaces, raw features are often too noisy for a classifier, but unsupervised scores distill that complexity into actionable signals.
  • Diversity Matters: The "Balance Selection" method highlights that an ensemble is only as good as its variety. Picking the five "best" unsupervised detectors often yields redundant information; picking five different types of detectors provides a more robust view of what constitutes an outlier.
  • Visualizing the Boost: Using t-SNE, the authors show that adding TOS significantly improves the separation between normal points and outliers compared to using original features alone.
t-SNE visualization on the Arrhythmia dataset showing how adding TOS helps separate outliers (red) from normal points (blue).

Key Results

  • Superior Accuracy: The full XGBOD ensemble (XGB_Comb) outperformed all baselines—including individual unsupervised detectors and previous semi-supervised methods like BORE—on 6 out of 7 datasets in both ROC and Precision@N metrics.
  • Significant Gains: On the Speech dataset, XGBOD improved Precision@N by over 23% compared to standard supervised XGBoost.
  • Efficiency: By replacing the costly "EasyEnsemble" method (used in prior work) with a single tuned XGBoost model, the authors achieved more stable results with faster execution times on large datasets like Mnist and Satellite.
The effect of the number of selected TOS on Precision@N across different datasets.

Limitations and Future Work

  • Computational Cost of Phase 1: Generating 100+ unsupervised scores for every data point can be computationally expensive, especially for algorithms with high time complexity like LoOP.
  • Heuristic Dependency: The performance of XGBOD is somewhat dependent on the initial pool of unsupervised functions. If the base detectors are poorly suited to the specific "pathology" of a dataset, the framework's gains may be limited.
  • Future Directions: The authors suggest exploring automated feature selection on the original data before generating TOS and investigating the use of dimensionality reduction techniques (like PCA) to combine unsupervised scores rather than just selecting them.

Summary updated

Abstract

A new semi-supervised ensemble algorithm called XGBOD (Extreme Gradient Boosting Outlier Detection) is proposed, described and demonstrated for the enhanced detection of outliers from normal observations in various practical datasets. The proposed framework combines the strengths of both supervised and unsupervised machine learning methods by creating a hybrid approach that exploits each of their individual performance capabilities in outlier detection. XGBOD uses multiple unsupervised outlier mining algorithms to extract useful representations from the underlying data that augment the predictive capabilities of an embedded supervised classifier on an improved feature space. The novel approach is shown to provide superior performance in comparison to competing individual detectors, the full ensemble and two existing representation learning based algorithms across seven outlier datasets.
BibTeX
@inproceedings{zhao2018xgbod,
  author = {Yue Zhao and Maciej K. Hryniewicki},
  title = {{XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning}},
  booktitle = {IJCNN},
  year = {2018},
  url = {https://openreview.net/forum?id=epghMdQOgs},
}

Loading related content...

Figure