TL;DR
Open-vocabulary segmentation models often lose their statistical reliability guarantees when new class names are added to the vocabulary after deployment. This paper introduces a "reference-anchored" conversion that makes class scores independent of other candidates, allowing a single calibration to provide valid risk bounds even as the vocabulary grows or changes per image.
What Problem Does This Paper Address?
Open-vocabulary segmenters are powerful because they can identify classes defined by arbitrary text at inference time. However, in safety-critical or high-precision applications (like robotics or medical imaging), we need more than just a "best guess"—we need a prediction set of plausible labels with a guaranteed "risk" level (e.g., "the true label is missing from this set less than 10% of the time").
This is typically handled via Conformal Risk Control (CRC), which calibrates a score threshold on a labeled dataset. The problem is candidate dependence: in most modern models (like CAT-Seg or FC-CLIP), the score for a specific label (e.g., "cat") changes depending on what other labels (e.g., "dog," "car") are in the current vocabulary. This happens because of mechanisms like softmax normalization or cross-label attention.
The authors show that when you calibrate a model on a small "source" vocabulary and then deploy it with a larger "grown" vocabulary, the risk guarantees break. On the ADE20K dataset, a model calibrated for 10% risk can see its actual risk jump to over 15% simply by adding more class names.
Methods
The authors propose a training-free solution called Reference-Anchored Conversion.
Instead of scoring a candidate label against the other labels in the requested vocabulary, the model scores each label against a fixed reference bank of names.
- For Transformer-based models (CAT-Seg, SED): The candidate label attends to the fixed bank and itself, but not to other requested candidates. The key-value sums of the bank are cached once per image, making the score for any label "candidate-independent."
- For Mask-based models (FC-CLIP): The softmax normalization is performed over the candidate plus the reference bank, rather than the shifting requested vocabulary.
Because the score for a label now stays the same regardless of what other labels are added, a threshold calibrated once on a source vocabulary remains valid for any superset of that vocabulary.
Datasets
The researchers evaluated their approach on three major benchmarks, simulating vocabulary growth:
- ADE20K: Growing from 150 benchmark classes to 845 classes.
- Pascal Context: Growing from 59 classes to 458 classes.
- COCO-Stuff: Growing from 171 classes to 1,283 classes (using LVIS categories).
- Reference Banks: Constructed using names from Open Images and the models' original training vocabularies.
Key Insights and Ideas
- The "Aha!" Moment on Dependence: The failure of conformal guarantees isn't due to the model getting "confused" by more classes, but due to the mathematical shift in scores caused by normalization and attention. By "anchoring" the context, you stabilize the statistics.
- Guarantees for Unseen Labels: The paper extends CRC theory to treat names as exchangeable units. If the source labels are a random sample of potential labels, you can actually provide a risk guarantee for labels the model has never seen annotations for.
- Efficiency Gains: In models like Grounding DINO, which typically process large vocabularies in chunks, anchoring allows the image-processing branch to run once against the bank rather than once per chunk. This resulted in a 6x inference speedup.
- Per-Image Vocabularies: The anchored guarantee holds even if the vocabulary is chosen dynamically for each image (e.g., by a Large Language Model describing the scene).
Key Results
- Reliability Restored: While "native" models consistently exceeded the target risk after vocabulary growth, the anchored models met the target risk (e.g., $\alpha = 10\%$) in every single test case.
- Minimal Overhead: To maintain the guarantee, anchored prediction sets are only slightly larger than a perfectly recalibrated native model—usually by less than one extra label per pixel.
- Accuracy Preservation: The conversion is training-free and results in minimal loss in segmentation accuracy (mIoU drops by at most 1.7 points).
- Speedup: Beyond reliability, the method made Grounding DINO significantly faster at scale.
Limitations and Future Work
- Bank Selection: While the risk guarantee holds for any bank chosen independently of the test data, the size of the prediction sets (efficiency) depends on the bank's quality. Finding the "optimal" bank remains an open question.
- Computational Cost: For some architectures, the anchored pass is more expensive than a single native pass because it processes the reference bank. However, the authors argue this is a worthwhile trade-off for the cost of constant recalibration.
- Softmax Attention: The authors note that their conversion covers linear attention; however, "softmax attention" across labels (where the denominator is a softmax over the vocabulary) remains an untested frontier for this specific anchoring technique.
Summary updated