Vision models routinely discover internal features nobody asked for: a direction in a sparse autoencoder (a model that rewrites an embedding as a short sum of reusable parts), an attention head, a latent slider in a generator. Before anyone can use these features, someone has to name them. The current way to automate naming is to ask language: score each candidate word by how well its text embedding aligns with the feature, or show top images to a multimodal language model and ask what they share. Language, however, is a skewed witness. People write "broken mirror" thousands of times and almost never write "rigid mirror," and they write "yellow" for a yellow car but not for a banana. A labeler built on caption statistics should therefore be reliable for features that people mention and unreliable for the mundane perceptual properties (material, rigidity, typical color, texture) that the vision encoder plainly sees but captions omit.
This note proposes to measure that failure and correct it. The main idea is to treat the disagreement between what an image shows and what its caption says as information. We separate "is this property visually present?" from "would a person bother to mention it here?", and we name each visual direction by its correlation with the first quantity only. The main prediction is specific. Existing automatic labelers lose accuracy as the true concept's caption-mention rate falls, while a presence-based labeler stays roughly flat. The same correction should apply when the directions live inside the projected visual tokens of a vision-language model, which is where the stakes for downstream reasoning are highest.
Core Claim
Text-aligned naming scores mix two factors. Following the factor-decoupling view of Seeing through the Human Reporting Bias, for an image $x$ and concept word $t$,
$$p(\text{mention } t \mid x) = p(\text{present } t \mid x)\, p(\text{relevant } t \mid \text{present}, x),$$
where "present" means the property is in the pixels and "relevant" means it is worth saying in that context. Contrastive image-text models are trained on captions, so their similarity scores estimate the left-hand side. A labeler built on them therefore down-weights concepts with low relevance, even when those concepts are visually present and strongly encoded. We propose naming a direction $d$ with the concept $t$ that maximizes the correlation, over a large image pool, between the direction's activation $a_d(x)$ and the estimated log presence $\log p(\text{present } t \mid x)$, rather than its correlation with mention or its embedding similarity to $t$. The reason this can work is that relevance depends on context ("yellow" is salient for a car but not for a banana), so it cannot be divided out by a single global frequency correction. It needs a model of relevance that is conditioned on the image.
Where This Sits in Prior Work
Three lines of work supply the method or motivate the gap. Unsupervised direction discovery in generators, including Householder Projector, NoiseCLR, DisDiff and WarpedGANSpace, finds clean directions but leaves naming to human inspection. That is the bottleneck we target, though we work with encoder directions where activation statistics are cheap to compute.
Two automatic labelers are our direct competitors. TEXTSPAN explains each CLIP attention head with a basis of ChatGPT-written descriptions. PCBM-ReD decomposes CLIP embeddings into sparse-autoencoder atoms and has a multimodal language model name them; its authors acknowledge that the model is weak on nuanced visual features. MM-CBM builds its concept vocabulary by prompting a language model. All three let language decide both which candidate words exist and how they are scored. We keep their decomposition machinery and replace the scoring.
The reporting-bias literature tells us where to expect failure. Do Neural Language Representations Learn Physical Commonsense? shows that text models get object–affordance pairs right but fail on affordance–property links, and that scaling the corpus does not help. PIQA shows a gap of roughly 20 points to humans that is driven by materials and physical manipulation. Seeing What Tastes Good finds that even human feature norms omit obvious perceptual traits. These results predict that perceptual properties are exactly where language-scored labels will be wrong.
The cross-modal conflict literature supplies the estimator. Self-MM derives unimodal pseudo-labels by shifting a shared label according to each modality's distance from class centroids, without new annotation. We reuse that move for concept presence. Starting from a caption-derived label, we shift it toward "present" when the visual feature sits near the present-centroid, which produces vision-only pseudo-labels. Where the caption and the visual estimate disagree, we have an omission. UniC and CH-SIMS show that text dominates consensus labels while other channels carry the missing signal. That is the affective analogue of our setting, and it constrains the design: a single shared label would wash the conflict out.
The closest bridging work, BiAug, fixes reporting bias by inpainting images and rewriting captions so that they match. It retrains the encoder and names nothing. We leave the encoder untouched and correct the interpretation of it.
Research Program
The first component is a presence model. On CLIP ViT-L image features we train the two-head decoupling model of Seeing through the Human Reporting Bias. One head predicts presence and the other predicts context-conditioned relevance, and both are trained from MSCOCO captions treated as noisy "mention" labels. The vocabulary is fixed: the 50 properties of the Situated physical-commonsense dataset, plus Visual Genome attributes. We do not use an LLM-generated pool.
We add Self-MM-style centroid-shifted pseudo-labels as the training signal for the presence head. We then compare that variant against the original latent-variable training. This comparison tests whether the conflict-derived pseudo-labels actually help.
The second component is the set of directions to name, drawn from three places. The first source is PCBM-ReD sparse-autoencoder atoms on CLIP embeddings. The second is TEXTSPAN head directions. The third is sparse-autoencoder atoms trained on the projected visual tokens of a LLaVA-style model, meaning the vectors after the small projector that maps vision features into the language model's word-embedding space. We train that model with frozen encoders and only the projector learned, following Scalable Vision Language Model Training.
The third source matters because a new labeler becomes available there: decoding a direction through the language model's own output-word layer. We expect this labeler to inherit the strongest reporting bias, because the language model's word prior is the caption prior. Deep Pre-Alignment for VLMs reports that projected tokens remain far from text space in early layers, so we read directions at several depths.
The third component is the naming rule. It chooses the concept $t$ that maximizes $\mathrm{corr}_x\big(a_d(x), \log p(\text{present } t \mid x)\big)$. The ablation replaces presence with mention, using the same correlation and the same vocabulary, so the only change is the decoupling.
The fourth component is a stratified evaluation set. For each direction, we take its top-activating and median images and compare them with human-verified property labels from Situated and Visual Genome. This gives a ground-truth concept for a subset of directions. Each such direction is tagged with the caption-mention rate of its true concept in COCO, measured conditional on that concept being present.
Baselines
The do-nothing reference is text-embedding similarity naming. Each direction gets the vocabulary word whose CLIP text embedding is most similar to it, which is what most practitioners do. It runs on the same directions and the same fixed vocabulary, and it has no tuning.
The strongest existing labelers are PCBM-ReD's multimodal-model naming and TEXTSPAN. They are the strongest because they are the published automatic labelers for exactly these directions and are built on the same CLIP backbone. On its own tasks TEXTSPAN reports a worst-group Waterbirds accuracy rise from 48% to 73% by ablating heads it named, and a zero-shot ImageNet-Segmentation mIoU of 54.50 against 47.47 for GradCAM. PCBM-ReD reports only a 0.41% average accuracy drop relative to a linear probe across 11 datasets. No reported result was found on Lacuna for naming accuracy itself under either method, so we will measure both on our stratified set.
To keep the comparison fair, both baselines get the same directions and the same image pool. Each is run twice, once with its native generated vocabulary and once restricted to our fixed property vocabulary, so that vocabulary and scoring effects can be separated. The multimodal labeler gets the same number of images per direction as our correlation estimate uses.
For vision-language models, the additional baseline is decoding a direction through the language model's output-word layer.
The key ablation is mention-correlation naming, which isolates the decoupling. A second ablation is global frequency normalization: divide each word's score by its average score over all images, a pointwise-mutual-information correction. This ablation isolates whether context-conditioned relevance matters at all.
Beating the baselines means two things. Top-1 naming accuracy must be higher in the lowest mention-rate tercile with non-overlapping bootstrap intervals. Accuracy in the highest tercile must be no worse. The difference in slope between accuracy and mention rate is the headline statistic.
Why not just…?
Why not just ask a strong multimodal model to look at the images and name the direction? It sees the pixels, so the objection is that it should not inherit caption bias. Its answer is still generated text, though, and generation favors salient words. PCBM-ReD's authors report exactly this weakness on nuanced features. This is a real threat for strong models, and the stratified curve settles it. If the multimodal labeler's accuracy does not fall with mention rate, the bias hypothesis is wrong for that labeler and we will report it as such.
Why not just give the labeler a fixed property checklist instead of letting it generate words? This may fix most of the problem, because some of the failure is that omitted properties never enter the candidate pool. The two-by-two design of vocabulary (generated or fixed) crossed with scoring (similarity or presence) measures this directly. If fixed vocabulary with similarity scoring closes the low-mention gap, the contribution shrinks to "use a feature-norm vocabulary," and we would say so plainly.
Why not just divide out global word frequency? It is one line of code, and it handles the "rigid mirror" case. It cannot handle context, as with "yellow" for a car versus a banana, because the same word is reported in one context and not the other. The frequency-normalization ablation tests this. If it matches the learned relevance head, we will adopt it as the method, since the simpler correction would be the better result.
Why not just debias the encoder with BiAug or NegCLIP and then use TEXTSPAN? That changes the model being explained, and inpainting is expensive by BiAug's own account. NegCLIP improves relation and order, raising VG-Relation from 63% to 81% and COCO-Order from 46% to 86% in When and Why Vision-Language Models Behave like Bags-Of-Words. It targets compositional structure, not omitted properties. We will still run TEXTSPAN on a NegCLIP model as a baseline at matched evaluation.
Decisive Checks
The first check is the stratified naming curve on CLIP and on projected vision-language tokens. The bias hypothesis predicts that the language-decoding labeler shows the steepest decline with mention rate.
The second check is causal. We ablate directions named for perceptual properties, in the style of TEXTSPAN's head mean-ablation, and measure the drop on ARO VG-Attribution for the attribute named versus control attributes. Neither ARO nor BiAug reports a VG-Attribution number on Lacuna, so we will report our own CLIP baseline. A correct label should produce a targeted drop. A mislabeled direction should not.
The third check is the downstream payoff. We take the affordance–property task of the Situated dataset, where BERT reaches 0.36 macro-F1 on affordances and 0.25 on properties against human scores of 0.65 and 0.67. A language model receives the presence-named properties active in the object's COCO image as text, compared with similarity-named properties. Gains there would show that naming what captions leave out supplies exactly the knowledge that text scaling does not.
References
- Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels
- Householder Projector for Unsupervised Latent Semantics Discovery
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models
- DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models
- WarpedGANSpace: Finding non-linear RBF paths in GAN latent space
- Interpreting CLIP's Image Representation via Text-Based Decomposition
- Concepts from Representations: Post-hoc Concept Bottleneck Models via Sparse Decomposition of Visual Representations
- Multimodal Concept Bottleneck Models
- Do Neural Language Representations Learn Physical Commonsense?
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
- UniC: a dataset for emotion analysis of videos with multimodal and unimodal labels
- CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
- Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association
- Scalable Vision Language Model Training via High Quality Data Curation
- Deep Pre-Alignment for VLMs
- When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?