Paired-Anchor Dictionaries: Cheap, Self-Labeled Concept Decomposition Across the Modality Gap

Vision–language models such as CLIP map images and captions into one shared vector space, but the two modalities never quite share it. Image vectors and text vectors sit in separate regions with an empty gap between them, called the modality gap. Recent work shows the gap has a clean internal structure. A handful of high-energy, modality-specific directions act as constant offsets, while a sparse set of shared directions carries the actual image–text matching. Remove the offsets and the gap closes with little loss in retrieval. Finding those directions currently takes a trained sparse autoencoder (a wide network that rewrites each vector as a few active "concept" units) and one to three million paired examples. It also takes a separate language-model pass to name each concept. This interpretability cost is exactly what the parameter-efficient multimodal literature works to avoid.

We propose to build the cross-modal concept dictionary without gradient training. Each dictionary entry is a real image–caption pair taken from the data. The dictionary grows greedily: a pair is added only when the current dictionary cannot jointly reconstruct its image and text vectors. Because every entry is an actual example, the caption serves as its label, so no automated labeling is needed. Because each entry has an image half and a text half, and entries may also be image-only or text-only, the split into shared and modality-specific directions falls out of construction instead of being imposed by a regularizer. The claim is that, on a data budget comparable to closed-form alignment methods (tens of thousands of pairs, minutes of compute), this dictionary matches trained multimodal autoencoders at the tasks they are used for: gap removal, retrieval, and gap-safe vector arithmetic. The claim can be checked directly against numbers those papers already report.

Core Claim

The method uses joint matching pursuit, a greedy procedure that explains a vector by repeatedly picking the dictionary entry most aligned with what remains unexplained. It runs over stacked image–text vectors, with a dictionary seeded from real pairs. The claim is that this recovers the unimodal-bias and bimodal-concept decomposition found by Cross-Modal Redundancy and the Geometry of Vision–Language Embeddings, using roughly 1–5% of that paper's 1M training pairs and no gradient steps. It also claims the bias part is richer than a single mean offset, which is why it beats the naive mean shift. What is new is the combination of three things: example-anchored atoms from inference-time decomposition, a paired-atom structure that forces one coefficient to explain both modalities, and explicit single-modality atoms that absorb the gap.

Relation to Prior Work

Three papers motivate the problem and set the targets. Cross-Modal Redundancy trains an aligned autoencoder (SAE-A) built on matching pursuit with an "iso-energy" penalty on 1M LAION pairs. It shows that pruning unimodal atoms keeps LAION retrieval recall at 0.760, against 0.704 for a centroid shift. VL-SAE learns a unified concept set with a contrastive pre-alignment stage and modality-specific decoders on CC3M. It uses this set to denoise OpenCLIP zero-shot classification and to reduce LLaVA hallucination. Universal Sparse Autoencoders aligns concepts across vision backbones through joint multi-decoder training. All three pay a full gradient-training cost and, apart from caption inspection, need separate labeling. We keep their evaluation targets and drop their training.

The method comes from ITDA. ITDA builds text-LLM dictionaries by adding real activations that fail to be reconstructed, which gives 100× faster construction and about 1% of the data. Its atoms are tied to specific prompts. ITDA works on a single modality and does not constrain codes across modalities. It also reports negative coefficients that muddy interpretation. We extend it to paired atoms and use a nonnegative variant. SAEBench constrains our evaluation choices: reconstruction quality is a poor proxy for usefulness, so we judge dictionaries by downstream effects and not by $R^2$ alone.

Two papers set the cost regime. CSA closes the gap between unimodal encoders with a closed-form correlation analysis on CPU, matching CLIP's ImageNet zero-shot accuracy from 35,000 pairs. Its projection is dense and uninterpretable, and it is the budget we aim to match. The stitching result of Transferring Linear Features Across Language Models, a 50% compute saving from warm-starting autoencoders, shows that interpretability cost can be amortized. However, its transfer stays within one model family. We do not attempt cross-modal stitching.

Research Program

Given frozen image and text embeddings $v_i, t_i \in \mathbb{R}^d$ from a dual encoder, we form stacked vectors $u_i = [v_i; t_i]$. There are three kinds of candidate atom:

  • paired atoms $[\hat v_a; \hat t_a]$, built from a real pair;
  • image-only atoms $[\hat v_a; 0]$;
  • text-only atoms $[0; \hat t_a]$.

Hats denote per-half unit normalization. Construction streams through pairs and codes each $u_i$ with nonnegative matching pursuit under a sparsity budget $k$. If the relative residual exceeds a threshold $\tau$, the method adds whichever candidate from that pair lowers the residual most. Encoding a lone image or caption at test time uses only the matching half of each atom.

The cross-modal constraint lives in the paired atoms. One coefficient must explain the image and the caption together. Directions that systematically differ between the modalities therefore cannot be absorbed by paired atoms and get pushed into single-modality atoms. Gap removal is then simple: re-encode each vector and drop the contributions of single-modality atoms. Each paired atom's label is its caption. Each single-modality atom is described by its anchoring example and by the inputs that fire it most.

We first run this on CLIP, OpenCLIP and SigLIP embeddings, the same model families used in Cross-Modal Redundancy. Construction data are LAION-400M subsets of 10k, 35k, 100k and 1M pairs. For each size we record wall-clock time and hardware. The second stage asks whether the method carries over to a generative multimodal LLM. There we take LLaVA 1.5 visual-token and text-token activations at one middle layer, build the dictionary from paired image/question activations, and use concept-level filtering of image-only atoms to test hallucination. This is the setting where VL-SAE reports its POPE gains.

Baselines

The naive reference points are the undecomposed embedding and the centroid shift, which subtracts the image mean minus the text mean from image vectors. Cross-Modal Redundancy reports LAION retrieval recall of 0.882 before any gap removal and 0.704 after the shift. On FashionIQ, the unrestricted arithmetic baseline has an out-of-distribution score of 0.97. We rerun both on our exact splits so the numbers are comparable.

The strongest competitor for gap decomposition is SAE-A itself. It reports recall of 0.760 after unimodal-atom pruning. Restricted to the bimodal subspace, it lowers FashionIQ out-of-distribution scores from 0.97 to 0.63 with recall of 0.683 against 0.681. We run it two ways: with its published 1M-pair budget, and data-matched on our smaller subsets with the same tuning sweep (dictionary size, sparsity, penalty weight). This separates "cheaper because less data" from "cheaper because no training." Beating it means that at 35k–100k pairs we match 0.760 recall within noise and reach an out-of-distribution score at or below 0.63, at a fraction of its wall-clock time.

For concept denoising and hallucination the strongest competitor is VL-SAE. It reports 14-dataset mean zero-shot accuracy on OpenCLIP ViT-H/14 rising from 76.9% to 77.8%, and LLaVA 1.5 POPE-Adversarial accuracy rising from 78.67% to 83.60%. We run it at its CC3M budget and data-matched. Beating it means recovering most of those gains, say at least 0.6 points of the 0.9-point zero-shot gain and at least 3 of the 4.9 POPE points, with no gradient training.

Two ablations isolate the key ingredient. The first is plain ITDA run separately per modality, with no paired atoms. The second is joint pursuit with paired atoms only, which removes the single-modality atoms. If either matches the full method, then pairing or explicit bias atoms are not doing the work.

Why not just…?

Why not just subtract the mean difference between modalities? It is the obvious zero-cost fix. The only evidence on it, from Cross-Modal Redundancy, shows it costs retrieval: 0.704 against 0.760 for atom pruning. That is evidence that the bias is spread over several directions, not one. Our method has to show the same advantage. If our single-modality atoms collapse to roughly one direction and our recall matches the shift, the premise fails and we will report that.

Why not just use the canonical-correlation projection of CSA, keeping high-correlation directions as "shared" and discarding the rest? This is the most serious threat. A correlation analysis already ranks directions by how much the two modalities agree, and CSA works from 35k pairs. We run it as a baseline in two forms: truncated to the top canonical subspace as a gap filter, and with our dictionary built inside the whitened canonical space. If the truncated projection matches our retrieval and arithmetic numbers, then the dictionary's contribution for gap removal is nil. What remains is monosemantic, self-labeled atoms, which a dense rotation cannot provide, and we would narrow the claim to that. CSA's own reported weakness on large dense retrieval suggests truncation may lose fine-grained detail. That is a hypothesis, not a result.

Why not just train VL-SAE or SAE-A on fewer pairs? Their costs may be dominated by data volume, which is easy to cut. The data-matched runs in our baselines answer this directly. The file contains no reported result for either at small budgets ("no reported result found on Lacuna" for sub-1M SAE-A or sub-CC3M VL-SAE). If they hold up at 35k pairs, the advantage shrinks to wall-clock time and free labels, and we will say so.

Decisive Experiments

The first check is gap decomposition on LAION subsets. We measure centroid gap, recall after dropping single-modality atoms, and dictionary-construction time, plotted against pair budget for all methods above. The success criterion is recall at or above 0.760 with fewer than 100k pairs.

The second is FashionIQ composed retrieval, restricting arithmetic to paired atoms. We report out-of-distribution score and recall against the 0.63/0.683 reference.

The third is the 14-dataset OpenCLIP zero-shot suite with concept-level denoising, measured against VL-SAE's 76.9%→77.8%.

The fourth tests whether the atoms mean what their captions say, which SAEBench warns reconstruction cannot show. We check label fidelity: whether held-out images that fire an atom have captions containing its anchor concept. We also run a disentanglement test on CelebA. There we build a split with a correlated attribute pair, ablate the atoms labeled with the spurious attribute, and measure the probe-accuracy change, in the spirit of spurious-correlation removal.

Finally, the LLaVA 1.5 stage reports POPE Random, Popular and Adversarial accuracy after suppressing image-only atoms, against 78.67%→83.60%. It also reports CHAIR, keeping in mind the instability documented by Evaluating Object Hallucination in Large Vision-Language Models (prompt standard deviation 3.22 against 0.78 for POPE). We therefore treat POPE as the primary hallucination measure.

Two failures would sink the proposal. The first is that the full method does no better than the paired-only or per-modality ablations. The second is that truncated canonical correlation matches it on every gap metric while also passing the label-fidelity test through post-hoc naming.