Calibrate Once, Grow the Vocabulary: Candidate-Independent Scores for Conformal Risk Control in Open-Vocabulary Segmentation

Md Tanveer Hossain Munim, Bijoy Ahmed Saiem, Muhammad Abdullah AdnanarXiv 2026 · arXiv

TL;DR

Open-vocabulary segmentation models often lose their statistical reliability guarantees when new class names are added to the vocabulary after deployment. This paper introduces a "reference-anchored" conversion that makes class scores independent of other candidates, allowing a single calibration to provide valid risk bounds even as the vocabulary grows or changes per image.

What Problem Does This Paper Address?

Open-vocabulary segmenters are powerful because they can identify classes defined by arbitrary text at inference time. However, in safety-critical or high-precision applications (like robotics or medical imaging), we need more than just a "best guess"—we need a prediction set of plausible labels with a guaranteed "risk" level (e.g., "the true label is missing from this set less than 10% of the time").

This is typically handled via Conformal Risk Control (CRC), which calibrates a score threshold on a labeled dataset. The problem is candidate dependence: in most modern models (like CAT-Seg or FC-CLIP), the score for a specific label (e.g., "cat") changes depending on what other labels (e.g., "dog," "car") are in the current vocabulary. This happens because of mechanisms like softmax normalization or cross-label attention.

The authors show that when you calibrate a model on a small "source" vocabulary and then deploy it with a larger "grown" vocabulary, the risk guarantees break. On the ADE20K dataset, a model calibrated for 10% risk can see its actual risk jump to over 15% simply by adding more class names.

Methods

The authors propose a training-free solution called Reference-Anchored Conversion.

Instead of scoring a candidate label against the other labels in the requested vocabulary, the model scores each label against a fixed reference bank of names.

  • For Transformer-based models (CAT-Seg, SED): The candidate label attends to the fixed bank and itself, but not to other requested candidates. The key-value sums of the bank are cached once per image, making the score for any label "candidate-independent."
  • For Mask-based models (FC-CLIP): The softmax normalization is performed over the candidate plus the reference bank, rather than the shifting requested vocabulary.

Because the score for a label now stays the same regardless of what other labels are added, a threshold calibrated once on a source vocabulary remains valid for any superset of that vocabulary.

Datasets

The researchers evaluated their approach on three major benchmarks, simulating vocabulary growth:

  • ADE20K: Growing from 150 benchmark classes to 845 classes.
  • Pascal Context: Growing from 59 classes to 458 classes.
  • COCO-Stuff: Growing from 171 classes to 1,283 classes (using LVIS categories).
  • Reference Banks: Constructed using names from Open Images and the models' original training vocabularies.

Key Insights and Ideas

  • The "Aha!" Moment on Dependence: The failure of conformal guarantees isn't due to the model getting "confused" by more classes, but due to the mathematical shift in scores caused by normalization and attention. By "anchoring" the context, you stabilize the statistics.
  • Guarantees for Unseen Labels: The paper extends CRC theory to treat names as exchangeable units. If the source labels are a random sample of potential labels, you can actually provide a risk guarantee for labels the model has never seen annotations for.
  • Efficiency Gains: In models like Grounding DINO, which typically process large vocabularies in chunks, anchoring allows the image-processing branch to run once against the bank rather than once per chunk. This resulted in a 6x inference speedup.
  • Per-Image Vocabularies: The anchored guarantee holds even if the vocabulary is chosen dynamically for each image (e.g., by a Large Language Model describing the scene).

Key Results

  • Reliability Restored: While "native" models consistently exceeded the target risk after vocabulary growth, the anchored models met the target risk (e.g., $\alpha = 10\%$) in every single test case.
  • Minimal Overhead: To maintain the guarantee, anchored prediction sets are only slightly larger than a perfectly recalibrated native model—usually by less than one extra label per pixel.
  • Accuracy Preservation: The conversion is training-free and results in minimal loss in segmentation accuracy (mIoU drops by at most 1.7 points).
  • Speedup: Beyond reliability, the method made Grounding DINO significantly faster at scale.

Limitations and Future Work

  • Bank Selection: While the risk guarantee holds for any bank chosen independently of the test data, the size of the prediction sets (efficiency) depends on the bank's quality. Finding the "optimal" bank remains an open question.
  • Computational Cost: For some architectures, the anchored pass is more expensive than a single native pass because it processes the reference bank. However, the authors argue this is a worthwhile trade-off for the cost of constant recalibration.
  • Softmax Attention: The authors note that their conversion covers linear attention; however, "softmax attention" across labels (where the denominator is a softmax over the vocabulary) remains an untested frontier for this specific anchoring technique.

Summary updated

Abstract

Open-vocabulary segmenters accept class names at inference. Conformal risk control can bound how often their prediction sets miss the true label, but we show that this bound breaks when names are added after calibration. On ADE20K, a threshold calibrated for 10% risk at 150 names reaches up to 15.4% risk at 845 names in released CAT-Seg models. The risk exceeds the target on every calibration split, yet the prediction sets grow far less than the target requires. In SED they instead grow larger than needed. The cause is that pruning, cross-label attention and vocabulary normalisation make each label's score depend on the other requested labels. Scores can move either way, so a threshold recalibrated at a larger vocabulary need not hold at smaller ones. We remove this dependence without training by scoring each label against a fixed reference bank of names. A label's score then stays the same when names are added, so one calibration keeps its guarantee as the vocabulary grows. The guarantee holds whatever names are added, even ones chosen per image after seeing it. On three benchmarks and five segmenters, the converted models meet the target at every vocabulary size we evaluate. For CAT-Seg and SED at alpha=10%, they use at most one more label per pixel than the released models recalibrated at the grown vocabulary. Treating names instead of images as calibration samples also bounds the miss rate of added names. If the annotated names are a random sample of the names in the calibration images, an added name's expected miss rate on those images is at most the target. For harder added names, the excess is at most their gap in mean miss rate. Applied to Grounding DINO, which splits large vocabularies into chunks, the conversion also makes inference about six times faster.
BibTeX
@misc{munim2026calibrate,
  author = {Md Tanveer Hossain Munim and Bijoy Ahmed Saiem and Muhammad Abdullah Adnan},
  title = {{Calibrate Once, Grow the Vocabulary: Candidate-Independent Scores for Conformal Risk Control in Open-Vocabulary Segmentation}},
  year = {2026},
  eprint = {2611.20715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20715},
}
Figure