Conformal Anomaly Detection with Guaranteed False Positive Rates

This direction is merged into another direction: conformal anomaly detection. Its last version stays readable and citable here.

Version 1 of 121 papers30 conceptsfrozen 2026-10-04

In anomaly detection, the most difficult question is rarely "How do we score this data point?" but rather "Where do we draw the line?" Whether you are using reconstruction error from an autoencoder, distance in a latent space, or density estimates, you are eventually left with a raw score. Without a principled way to set a threshold, these scores are just arbitrary numbers. If you set the threshold too low, you are overwhelmed by false alarms; too high, and you miss the very threats you are trying to detect.

A growing body of research has converged on Conformal Prediction (CP) as the solution to this calibration problem. By wrapping traditional anomaly detectors in a conformal framework, researchers are transforming heuristic "weirdness" scores into statistically rigorous p-values that come with a mathematical guarantee: the false positive rate will not exceed a user-defined threshold.

The Calibration Gap

The fundamental problem in unsupervised detection is "hardness bias." As Adam Goodge and Bryan Hooi point out in their work on CADET, some normal data points are simply harder to model than others. A complex but benign image might yield a higher error score than a simple but anomalous one.

Standard methods treat every high score as a potential anomaly. Conformal anomaly detection shifts the perspective: it asks how "surprising" a new score is compared to a held-out calibration set of known normal examples.

How Conformalization Works

The process, often referred to as Inductive Conformal Prediction, follows a simple but powerful logic:

  1. Score: Take a pre-trained model and calculate a "non-conformity score" (like a distance or error metric) for every point in a clean calibration set.
  2. Rank: When a new test point arrives, calculate its score.
  3. Quantify: Determine where this new score ranks relative to the calibration scores. This rank is converted into a p-value.
  4. Threshold: If the p-value is smaller than a significance level $\alpha$ (e.g., 0.05), the point is flagged.

As Evgeny Burnaev and Vladislav Ishimtsev demonstrated in their early work on time-series, this procedure ensures that the probability of a normal observation being flagged is at most $\alpha$. This moves the reliability of the system from empirical guesswork to mathematical certainty.

From Social Networks to Robotics

This "statistical smoke detector" is being applied across diverse domains where the cost of a false alarm is high:

  • Network Security: Rui Luo and Vikram Krishnamurthy applied these principles to social networks, using the concept of "edge exchangeability" to bound false positives in fraud detection.
  • Time-Series and OOD: Frameworks like CODiT and iDECODe, developed by Ramneet Kaur, Sangdon Park, and Susmit Jha, use conformal prediction to detect when a model's "temporal common sense" fails, providing safety guarantees for autonomous systems.
  • Robotics: In the SAFE framework, Qiao Gu and colleagues use functional conformal prediction to create time-varying thresholds, allowing robots to detect task failure without constantly "crying wolf" during complex maneuvers.

Defending Against Malicious Attacks

Conformalization is also proving vital in adversarial contexts. Xun Xian and Mingyi Hong utilized it to build backdoor defenses that guarantee a limit on how often benign inputs are misidentified as malicious. Similarly, Zhen Xiang and Bo Li introduced CBD, the first framework to provide theoretical guarantees for detecting backdoored models themselves.

Even in large-scale production, the need for rigor remains. Google DeepMind’s SynthID-Image, detailed by Rudy Bunel and David Stutz, uses conformal prediction to maintain strict false-positive guarantees when detecting AI-generated watermarks across billions of images.

Why It Works: The Power of Exchangeability

The magic of conformal anomaly detection lies in its distribution-free nature. It doesn't matter if your anomaly scores follow a Gaussian distribution or something far more exotic. As long as the data is exchangeable (meaning the joint probability doesn't change if you swap the order of observations), the mathematical guarantees hold.

By shifting the focus from the magnitude of an error to its statistical significance relative to a baseline, conformal methods allow practitioners to deploy anomaly detectors with a known "error budget." As Sebastian Jäger and Felix Biessmann show in their work on Conformal Data Cleaning, this turns "black-box" models into calibrated tools that can automatically flag and repair data errors with a user-defined confidence level.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_conformal_anomaly_detection_with_guarant_12766590,
  title = {Conformal Anomaly Detection with Guaranteed False Positive Rates},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/conformal-anomaly-detection-with-guaranteed-false-positive-rates/rdr_12766590bd4d48219e9bb543e22230f9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_conformal_anomaly_detection_with_guarant_12766590_v1,
  title = {Conformal Anomaly Detection with Guaranteed False Positive Rates},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/conformal-anomaly-detection-with-guaranteed-false-positive-rates/rdr_12766590bd4d48219e9bb543e22230f9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}