Calibrating Rejection Thresholds Without Unseen Data

41 concepts35 papers2 proposals

When a standard machine learning model encounters an input it has never seen before, it rarely admits ignorance. Instead, it typically forces the novel input into a known category, often with dangerously high confidence. To prevent this, open-set recognition (OSR), out-of-distribution (OOD) detection, and zero-shot learning rely on a rejection threshold. If a sample's similarity, confidence, or distance score falls past this boundary, it is flagged as "unknown."

But here lies a fundamental paradox: How do you set a precise decision boundary for data you have never observed?

If you draw the boundary too tightly around your known data, the model becomes overly sensitive, rejecting valid variations of known classes. If you draw it too loosely, the model becomes overly permissive, letting novel anomalies slip through. Without access to examples of the "unknown" during training, choosing this threshold has historically been a matter of arbitrary guesswork.

The Fragility of Static Boundaries

In a traditional closed-world setting, thresholds are calibrated using a validation set containing examples of every target class. However, when moving to open environments, this assumption collapses.

As Schölkopf et al. noted in their foundational work on One-Class SVMs, adjusting the decision boundary to ensure robust generalization across unseen distributions remains an open challenge. Kassab similarly highlighted that establishing automated thresholding strategies that generalize across varying data distributions is highly difficult due to the unpredictable nature of noisy, real-world data.

When models deploy into production, they face an acute trade-off between false positives and false negatives. As detailed by Jin et al. and Gao et al., the effectiveness of open-set rejection mechanisms hinges entirely on this similarity threshold. Because the statistical properties of unseen classes are completely missing during training, thresholds optimized on known validation data frequently fail to generalize when the model encounters truly novel out-of-distribution samples in the wild (Roitberg et al.; McCarthy et al.).

[ Known Data Cluster ]  --->  Where do you draw the line?
       (  *   *  )
      (  *  *  *  )      ---?--->  [ Tight Threshold: High False Rejections ]
       (  *   *  )       ---?--->  [ Loose Threshold: High Misidentifications ]

                                   [ ? ] Unseen Novel Class (Position Unknown)

The Zero-Shot and Domain Shift Extremes

The threshold calibration problem becomes even more severe in zero-shot learning and unsupervised domain adaptation, where models must evaluate data under significant distribution shifts without any local ground truth.

In zero-shot text or event classification, models map inputs to entirely unseen categories based purely on semantic descriptions or definitions. Without a labeled validation set to anchor these boundaries, the model's performance becomes highly sensitive to arbitrary parameter selection (Sappadla et al.; Yu). Historically, systems have resorted to rigid workarounds, such as using fixed false-alarm rates derived from background data (Natarajan et al.), or relying on fragile tree structures that fail to generalize if the operational environment shifts (Liu).

Similarly, when models transition across domains, a universal constant threshold fails because it is impossible to pre-measure the test distribution in real time (Long et al.). In unsupervised target domains or single-domain generalization settings, the lack of ground-truth reference data makes it incredibly difficult to draw a clean line between known and unknown categories (Fu et al.; Jing et al.; Zhu et al.). The same challenge appears in security contexts like membership inference, where complex feature boundaries make unsupervised threshold selection highly unpredictable (Bai et al.).

Emerging Strategies for Unsupervised Calibration

To move away from manual tuning and arbitrary heuristics, recent research focuses on dynamically inferring the boundaries of the known space or bypassing the isolation of single-sample thresholds entirely.

Data-Driven Calibration Networks

Instead of guessing a static threshold, some frameworks introduce dedicated modules to automatically adjust boundaries based on data density. For instance, the authors of Few-Shot Open-Set Recognition... utilize an Outlier Calibration Network to eliminate manual thresholding entirely, even in low-data regimes. Similarly, Cao et al. introduce data-driven heuristics that adaptively flag unknown inputs for clinical deployment, reducing reliance on prior knowledge of anomaly rates.

Collective Decision Making

Rather than evaluating each test sample in isolation—which forces a hard threshold decision on every single input—Geng et al. proposed evaluating test data collectively in batches. By using a modified Hierarchical Dirichlet Process to cluster the incoming test stream, the model can discover entirely new categories simultaneously, bypassing the need for empirical thresholds altogether.

Background Regularization and Sample-Free Bounds

Another approach is to explicitly define what the "unknown" looks like during training without explicitly collecting target anomalies. Choo et al. use background-class regularization to create a statistical "safety zone" around known classes, giving the model a grounded way to set margins.

Alternatively, if no external data is available, models can look inward. Begon et al. demonstrate that models can leverage internal "white-box" statistics—such as batch normalization layers and latent space geometry—to rank deviations and establish sample-free boundaries without needing explicit anomaly examples during training.

Key Foundational Work

Incremental data-driven learning of a novelty detection model for one-class classification with application to high-dimensional noisy data (2008) — contributed 1 concept elements to this direction.
Support Vector Method for Novelty Detection (1999) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unsupervised_threshold_selection_44cf2224,
  title = {unsupervised threshold selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-threshold-selection/rdr_44cf22247f5748b6aa6ce1cd2392a233},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unsupervised_threshold_selection_44cf2224_v1,
  title = {unsupervised threshold selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-threshold-selection/rdr_44cf22247f5748b6aa6ce1cd2392a233/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake