Estimating Outlier Proportions in Unsupervised Anomaly Detection

This direction is split into other directions: outlier ratio estimation. Its last version stays readable and citable here.

Version 1 of 161 papers79 conceptsfrozen 2026-10-04

In unsupervised anomaly detection, the goal is to identify "atypical" objects without the help of ground-truth labels. Most algorithms, from classic Isolation Forests to modern deep autoencoders, excel at producing an anomaly score—a continuous value representing how "weird" a data point looks.

However, a score is not a decision. To actually flag an engine as failing or a transaction as fraudulent, a practitioner must set a threshold. This threshold is fundamentally tied to the contamination factor ($\gamma$): the proportion of anomalies present in the dataset. As Lorenzo Perini, Jesse Davis, and others have noted, this factor is almost always unknown in real-world scenarios. If you guess too high, you suffer from excessive false alarms; guess too low, and you miss critical failures.

The Hidden Hyperparameter Problem

Traditionally, researchers treated the contamination factor as a fixed hyperparameter, often defaulting to an arbitrary value like 5% or 10%. But as Daochen Zha and colleagues point out in their work on PyODDS, treating this ratio as a constant leads to suboptimal results because the true outlier rate varies wildly across domains.

The challenge is circular: to find the anomalies, you need the threshold; to set the threshold, you need to know how many anomalies there are. This has led to a new research direction focused on automated contamination estimation, moving the field toward "truly unsupervised" deployment.

Bayesian Estimation of the Contamination Distribution

One of the most precise ways to break this circle is to treat the contamination factor not as a single number, but as a probability distribution. Lorenzo Perini, Paul-Christian Bürkner, and Arto Klami introduced $\gamma$GMM, a Bayesian framework that estimates this distribution without using any labels.

The intuition behind $\gamma$GMM is that while a single anomaly detector might be noisy, the consensus across multiple detectors reveals the underlying data structure. By modeling the scores from an ensemble of detectors as a mixture of "normal" and "anomalous" distributions, the framework can infer the most likely proportion of outliers. This allows practitioners to quantify their uncertainty—knowing not just where the threshold should be, but how confident the model is in that choice.

Iterative Refinement and Self-Cleaning

Another approach to the contamination problem involves iterative refinement. In many cases, the presence of outliers in the training set actually "pollutes" the model, making it learn that anomalies are normal.

To combat this, Minkyung Kim and Tae-Hyun Oh proposed Iterative Anomaly Detection (IAD). Instead of requiring a pre-defined ratio, IAD uses sample-wise weights to progressively "ignore" points that look like outliers. As the model gets better at defining the "normal" manifold, it becomes more confident in which points to exclude, effectively cleaning its own training data.

Similarly, Boyang Liu and Jiayu Zhou developed RCA (Robust Collaborative Autoencoders), where multiple models peer-filter each other. By only updating weights on samples that the "peer" model deems normal, the system prevents the "hypersphere collapse" or memorization of outliers that plagues standard deep learning approaches.

Transferring and Sampling the Ratio

When direct estimation is too difficult, researchers look for "anchors" in other data.

  • Shape Similarity: Lorenzo Perini and Vincent Vercruyssen developed TRADE, which transfers a known contamination factor from a source domain to a target domain by matching the geometric shapes of their score distributions.
  • Importance Sampling: In semi-supervised settings, Aodong Li and Marius Kloft showed in their work on SOEL that you can estimate the global contamination ratio from a very small, biased set of labeled queries. By using importance sampling to correct for selection bias, they turn a few expert labels into a highly accurate estimate of the entire dataset's purity.

Why This Works: The Geometry of "Normal"

The fundamental reason these methods work is that "normal" data usually occupies a dense, low-dimensional manifold, while anomalies are scattered and "atypical." Whether through Kernel Fisher Discriminants as explored by Volker Roth, or Support Vector Novelty Detection by Bernhard Schölkopf, the goal is to find the boundary of this manifold.

By automating the estimation of the contamination factor, the field is moving away from "guessing" the boundary and toward discovering it. This transition is essential for deploying anomaly detection in autonomous systems—like jet engine monitoring or real-time cybersecurity—where the "normal" state is constantly shifting and human intervention is impossible.

Papers in this version

Showing 60 of 61 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_estimating_outlier_proportions_in_unsupe_3ad10f2a,
  title = {Estimating Outlier Proportions in Unsupervised Anomaly Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/estimating-outlier-proportions-in-unsupervised-anomaly-detection/rdr_3ad10f2a3ce54be5817549f350f79505},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_estimating_outlier_proportions_in_unsupe_3ad10f2a_v1,
  title = {Estimating Outlier Proportions in Unsupervised Anomaly Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/estimating-outlier-proportions-in-unsupervised-anomaly-detection/rdr_3ad10f2a3ce54be5817549f350f79505/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}