Text Alone Predicts Which Classes a Segmentation Readout Keeps

David YanarXiv 2026 · arXiv

TL;DR

Open-vocabulary segmentation models often "silently" lose the ability to recognize new classes that are semantically distant from their training data. This paper demonstrates that this "vocabulary-support collapse" is entirely predictable from text alone—before the model even sees an image—and can be reversed by simply turning down the "amplitude" of the trained readout head.

What Problem Does This Paper Address?

Open-vocabulary segmentation (OVS) is supposed to be the "Swiss Army knife" of computer vision, capable of segmenting any object a user names (e.g., "ottoman," "pool table," "runway"). However, to improve performance, researchers often train a "readout head" on top of frozen, pretrained features (like CLIP).

The authors discover a critical flaw: this training process causes the model to "collapse" on classes that weren't in the training set. Specifically, if a deployment class name is semantically "far" from the training vocabulary, the model will never assign it a single pixel, even if it is highly confident. Worse, standard metrics like mean Intersection-over-Union (mIoU) can actually increase while these classes are dying, making the failure invisible to developers.

Methods

The researchers propose the Vocabulary-Distance Mechanism (VDM) to explain and predict this collapse.

  1. Predicting Collapse: They show that a simple text similarity metric—the cosine similarity of a class name to its nearest neighbor in the training vocabulary ($s_c$)—is a powerful predictor of whether that class will survive.
  2. The Audit Protocol (KeepCert): They introduce a protocol to "audit" a model before deployment. By ranking classes by $s_c$, they can identify a "break-even" point ($s^*$). Classes below this point are at high risk of collapse.
  3. The Amplitude Knob: They identify that the "dose" of training is the culprit. By treating the learned part of the readout as a residual, they can control its "amplitude." Reducing this amplitude (or stopping training very early) prevents the learned weights from "outvoting" the broad knowledge held in the frozen features.
Figure 1. The list alone says which classes die. ADE-150 classes sorted by text similarity to the training list; below the break-even point they win no pixels (red), and turning the readout’s amplitude down brings them back.

Datasets

The models were trained primarily on COCO-Stuff-171. Evaluation was conducted across a wide range of deployment vocabularies and benchmarks, including:

  • ADE20K (ADE-150)
  • Pascal Context (PC-59 and PC-459)
  • PASCAL VOC (PAS-20)
  • A-847
  • OpenBench
  • LVIS-1203

The study spanned nine model families, twelve backbones (including TIPS, CLIP, and ConvNeXt), and eight pretraining recipes.

Key Insights and Ideas

  • Text is Destiny: The geometry of the text embedding space determines which classes a model will keep. You can predict a model's blind spots just by looking at two text files: the training list and the deployment list.
  • The "Help-or-Hurt" Flip: The training vocabulary is a causal intervention. Moving a single class name into or out of the training list can swing that class's performance by over 20 IoU points, even when the training images remain identical.
  • Features Survive, Readouts Kill: The underlying frozen features (like CLIP) actually contain the information needed to recognize the "collapsed" classes. The trained readout head simply learns to ignore that information in favor of training-set neighbors.
  • Specialization is the Enemy of Breadth: Training the readout is a form of immediate specialization. As the model gets better at the training classes, it rapidly "closes the cap" on what else it can see.
Figure 3. Text alone separates kept from collapsed.

Key Results

  • High Predictive Power: Text similarity ($s_c$) ranks collapse risk with an AUROC of 0.69–0.96 across various model architectures.
  • Massive Recovery: On the ADE-150 dataset, 51 of 150 classes collapsed (won zero pixels) after full training. Simply stopping training after one epoch (early stopping) recovered +16.2 IoU for these "far" classes.
  • Zero-Shot Repair: Turning down the readout's amplitude post-hoc (without any retraining or new labels) recovered +15.3 IoU on ADE-150.
  • Predictable Break-even: The model's "break-even" point $s^*$ can be predicted from the training list's own internal clustering to within 0.013 on average.
Figure 6. One dose axis, two views. (a) Training loss falls while the far band’s own loss, never optimized, rises. (b) At evaluation the same axis reads as the residual’s amplitude. The far band dies along it...

Limitations and Future Work

  • Transferability: While highly robust, the prediction failed to transfer perfectly in the LVIS-1203 benchmark, suggesting that within-benchmark superiority doesn't always translate to absolute cross-dataset performance.
  • Scope: The study focuses on the "frozen-backbone, trained-readout" regime. It remains to be seen how these findings apply to models where the entire backbone is fine-tuned.
  • Certification Costs: While collapse is predictable from text, providing a formal "certificate" of performance still requires a small labeled slice of data for calibration.

The authors suggest that future work should treat the training vocabulary as a primary design choice, rather than a static dataset property, to ensure models remain truly "open" after deployment.

Summary updated

Abstract

An open-vocabulary segmentation readout silently loses a class deployed beyond its training categories. It still emits masks with high confidence, but the class never wins a single pixel. The training loss falls anyway, and mIoU can even rise while classes die. We show that which classes collapse is written in the class-name list alone. Before any image is seen, we rank each class's collapse risk from its text similarity to the training vocabulary alone. The ranking holds at AUROC 0.69-0.96 across nine model families, twelve backbones and eight pretraining recipes. Two further families, measured only after the predictions were written down, land in the same band. What the list writes, it can rewrite. With identical images and a class's counterpart moved in or out of the training list, retraining flips that class's gain over the frozen baseline by 21 IoU points; swapping in unrelated names changes nothing. We find a vocabulary-distance mechanism behind the collapse. Collapsed classes lose every vote to near-name neighbors, but in the frozen features they survive. Turning down the readout's amplitude, or stopping training early, brings them back, up to +16 points, without retraining; trained with its amplitude held down, the readout never collapses.
BibTeX
@misc{yan2026text,
  author = {David Yan},
  title = {{Text Alone Predicts Which Classes a Segmentation Readout Keeps}},
  year = {2026},
  eprint = {2611.11926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11926},
}
Figure