Peeling Away the Class: Residual Topic Discovery for Unlabeled VLM Shortcuts

Vision-language models such as CLIP, which place images and text in one shared embedding space, now sit under many downstream systems: detectors, segmenters, image editors, concept explainers. Those systems inherit CLIP's shortcuts. A bird classifier built on CLIP may rely on whether the background is water, and a face-attribute classifier may rely on gender. Finding these shortcuts usually takes group labels, an object detector, or a list of suspected confounders written in advance. We propose to find them with a topic model instead. The specific move is borrowed from Semantic Component Analysis, a text topic model. Within each class, first remove the direction that the class itself occupies in CLIP space. Then cluster what is left. The leftover clusters are the strongest non-class themes, such as "water," "night," or "woman." Each one can be named by matching it against a vocabulary through CLIP's text encoder.

The output is a named, multi-level map of each class's hidden subpopulations, produced with no group annotations. We then use this map in two ways. As an audit, it says which confounders a CLIP-based model is exposed to. As a substitute for the group-annotated validation sets that current debiasing methods still need, it supplies pseudo-groups for model selection and reweighting. The claim that can be tested is that clustering the class-removed residual recovers true spurious groups better than clustering raw embeddings. It should also recover a second, weaker confounder when one exists, and existing single-pass clustering misses that second confounder.

Core Claim

Semantic Component Analysis (SCA) treats a text embedding as a sum of topic directions. It clusters the embeddings, subtracts each cluster's mean direction, and re-clusters the remainder. On text, this exposed secondary topics that the dominant theme had hidden: 182 topics versus BERTopic's 55 on Trump Tweets, with near-zero documents left as noise. SCA has only been run on sentence-encoder text vectors.

Our claim is that the same peeling becomes a shortcut detector when it is applied within a labeled class of CLIP image embeddings. Here the dominant component is known in advance: it is the class. The structure that remains after removing it is, by construction, whatever varies within the class, and on spurious-correlation benchmarks that variation is largely the confounder.

For an image embedding $x$ with class label $y$, let $c_y$ be the unit-normalized mean embedding of class $y$. The first-level residual is

$$r^{(1)} = x - (x^\top c_y)\, c_y .$$

We cluster $r^{(1)}$ with UMAP (a nonlinear method that compresses vectors to a few dimensions while keeping neighborhoods intact) followed by HDBSCAN (a density clusterer that may label points as noise rather than forcing them into a cluster). This yields first-level topics $k$ with centroids $s_k$. We then peel again, $r^{(2)} = r^{(1)} - (r^{(1)\top} s_k) s_k$, and re-cluster to find a second confounder. Benchmarks such as UrbanCars, where vehicle type is tied to both background and co-occurring objects, need this second level.

Each topic is named by scoring a noun-and-phrase vocabulary against the topic's residual centroid with CLIP's text encoder, contrasting it against the other topics in the same class. This is the reverse-classification idea of Image Clustering with External Guidance, adapted so that names are contrastive across topics in the spirit of class-based TF-IDF.

One objection needs answering directly. SEM shows that orthogonal projection badly damages content when it is used to debias embeddings (content preservation 0.415 versus 0.878 for its sparse method). Our projection is used only to discover groups, never to change the representation a classifier sees. The content loss that SEM measures therefore does not apply to the downstream model.

Relation to Prior Work

Three lines of work motivate this proposal. First, What do we learn from inverting CLIP models? inverts CLIP embeddings and finds strong social skew, for example "firefighter" rendering male 99% of the time. Second, Contrastive Adapters for Foundation Model Group Robustness trains small adapters on frozen CLIP and reports average-versus-worst-group gaps of up to 80.7 points. Third, Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning fine-tunes CLIP to push class text away from spurious attribute text. Together they establish that the shortcuts are real and costly.

The same work also constrains the design. The Yang et al. method finds shortcuts with an OWL-ViT detector, so it can only see detectable objects and not lighting, texture, or demographic attributes. The contrastive adapters still rely on group-labeled validation data for model selection. Our method needs neither.

The closest competitor is CoBalT, from Unsupervised Concept Discovery Mitigates Spurious Correlations. It discovers concepts without supervision and keeps 89.2% worst-group accuracy on CelebA without validation group labels. Our proposal differs in three ways. Discovery happens in the class-removed residual rather than the raw space. It is hierarchical, so it can find more than one confounder per class. And each group comes with a human-readable name, which turns the procedure into an audit tool and not just a training trick.

On the topic-modeling side, CEMTM and the CLIP-based models in Neural Multimodal Topic Modeling: A Comprehensive Evaluation treat the vision-language model as a trustworthy source of meaning. We instead treat its geometry as the object being audited.

Research Program

We first build the peeling pipeline on frozen CLIP (ViT-B/32 and ViT-L/14) image embeddings, with a fixed vocabulary of WordNet nouns plus attribute adjectives for naming. The first question is diagnostic. On benchmarks with known confounders, do the residual topics line up with the true spurious attribute, and does the top-ranked name identify it?

The second question is practical. We replace ground-truth groups with the discovered topics, crossed with class labels, and use them in three places: worst-pseudo-group accuracy for early stopping, group-balanced reweighting of a linear probe, and model selection for a contrastive adapter trained as in Zhang and Ré.

The third question is whether the second peeling level adds anything. UrbanCars is the test case, because it has two co-occurring confounders.

Finally, as a short secondary study, we run the same within-topic peeling on MS COCO and T4SA multimodal topics. The aim is to check whether CLIP-based topic models produce topics whose second-level structure is demographic or attribute-prior driven, which would be an instance of the binding bias that Magnet traces to CLIP's text encoder. This study is exploratory and is not part of the main claim.

Baselines

The naive reference point is empirical risk minimization (ERM), meaning a linear probe on frozen CLIP features trained to minimize average loss. NeuronTune reports 72.6% worst-group accuracy (WGA) for its ERM baseline on Waterbirds.

The strongest existing group-label-free method is CoBalT, which reports roughly 89–90.6% WGA on Waterbirds, 89.2% on CelebA without validation group labels, and 80.0% on UrbanCars (versus 76.9% for SPARE). NeuronTune reports 92.2% WGA on Waterbirds. Contrastive adapters report 90.0% WGA on CelebA, but with group-labeled validation. These numbers come from different backbones and training protocols, so we will not compare against them as printed. Instead we re-implement CoBalT and NeuronTune on the same frozen CLIP features, the same train/validation splits, and the same budget of 30 hyperparameter trials. All methods select hyperparameters using only pseudo-groups or average accuracy; none sees ground-truth group labels.

Beating the baseline means a higher WGA than the re-run CoBalT on at least two of the three benchmarks at equal average accuracy. On UrbanCars specifically, it means a gain that comes from the second peeling level.

The key ablation isolates the peeling step. It runs the same UMAP/HDBSCAN pipeline, the same naming, and the same downstream use, but clusters raw within-class embeddings with no class direction removed. A further ablation replaces HDBSCAN with K-means at the true group count. That isolates whether density clustering's noise handling matters.

For discovery quality, we compare against an OWL-ViT detection-discrepancy pipeline in the style of Yang et al. This comparison is measured by agreement between discovered and true spurious attributes, not by WGA.

Why not just…?

Why not just run K-means on raw CLIP embeddings within each class? This is the most serious threat, because on Waterbirds the background may already dominate within-class variance. The evidence against it is indirect. Image Clustering with External Guidance reports only 38.1% accuracy for raw-CLIP K-means on ImageNet-Dogs, and SCA shows that single-pass clustering hides secondary themes. Neither result is about shortcuts, though. The raw-clustering ablation settles the question. If it matches residual peeling on group-recovery NMI (normalized mutual information) and on WGA, then peeling is not the contribution. In that case we would narrow the claim to the multi-level UrbanCars setting and to the naming component.

Why not just ask a captioning model or an LLM to list the shortcuts? Prompting gives names but no assignment of individual images to groups, and downstream reweighting needs those assignments. SCA also outperformed LLM-based TopicGPT on US Bills while running on a CPU in minutes. As a check, we will caption a sample of images per class, have an LLM propose confounders, assign images by CLIP similarity to those proposals, and compare group-recovery NMI against residual topics.

Why not just use sparse autoencoder features to find the confounders? SEM and MICLIP show that sparse autoencoders, which re-express a dense embedding as many rarely-active features, separate entangled directions. Ranking those features by within-class variance is a credible alternative. We include it as a discovery baseline under the same naming and downstream protocol. If it wins, residual peeling is the cheaper but weaker option, and we will report that.

Why not just use an object detector? Detector pipelines can only find confounders that are objects. CelebA's gender confound and UrbanCars-style backgrounds are not objects in that sense, so the detector-based comparison above should show the gap directly.

Proposed Experiments

The benchmarks are Waterbirds, CelebA, and UrbanCars, all of which have ground-truth spurious attributes. We measure discovery with three metrics. The first is NMI and ARI (adjusted Rand index) between residual topics and the true confounder within each class. The second is naming precision, meaning whether the top-ranked term matches the confounder. The third is the noise rate. Downstream, we measure WGA and average accuracy with ground-truth group labels withheld everywhere except final evaluation.

The decisive check is the gap between the full method and the raw-clustering ablation on UrbanCars second-level recovery. The full method should recover the co-occurring-object confounder that a single-pass method misses. If it does not, the hierarchical claim fails.

A second decisive check is the correlation between worst-pseudo-group accuracy and true WGA across training checkpoints. A high correlation would show that discovered topics can replace group-labeled validation sets, the dependency that adapter methods still carry.

The exploratory COCO/T4SA audit is reported qualitatively, together with IEC (image embedding-based coherence), the image-coherence metric of the multimodal topic-modeling evaluation, which correlates 0.45 with human ratings. We will check whether second-level topics inside caption topics split along gender or color priors.