Behaviorally Silent Dimensions: The Modality Gap as a Source of Representation–Behavior Divergence

Image-text models such as CLIP make decisions in a specific way. To classify a photo "zero-shot", without training a new classifier, the model compares the photo's embedding with the embeddings of text prompts like "a photo of a dog" and picks the closest one. The model's behavior therefore depends only on the parts of the image embedding that line up with text embeddings. Studies that ask whether these models resemble human brains or human similarity judgments usually measure something else: the geometry of the full image embedding. It is well documented that image and text embeddings occupy separate, narrow regions of the shared space, a separation called the modality gap. That makes it likely that much of the image embedding lies in directions the text side never reads. Those directions can make a model look brain-like or human-like without having any effect on what it decides.

We propose to measure this directly. We split each image embedding into a "readout" part, the subspace that text prompts can see, and a "silent" remainder. We then ask three questions. How much of a model's measured alignment with human fMRI and human similarity judgments sits in each part? Do known interventions that change human-likeness act on the part that matters for the claim being made? Does alignment measured only in the readout subspace predict human-like behavior across models better than standard full-embedding alignment does? If the answers are "a lot sits in the silent part", "often not", and "yes", then part of the widely reported mismatch between representational and behavioral alignment is a geometric consequence of the modality gap, and it can be corrected with a cheap readout-aware measurement.

Core Claim

In dual-encoder contrastive models, the subspace that zero-shot behavior depends on is low-dimensional and partly misaligned with the image representation, because of the cone geometry and the separating gap direction. Alignment metrics computed on full image embeddings therefore mix behaviorally effective structure with behaviorally silent structure. We claim that this mixing explains a measurable share of the dissociation between neural and behavioral alignment. We also claim that restricting representational metrics to the readout subspace, or editing only that subspace, makes representation and behavior move together again.

The motivating literature is clear that representation and behavior come apart. How Aligned are Different Alignment Metrics? analyzes 241 models and finds that neural and behavioral metrics have an average pairwise rank correlation of only 0.198. Cheung et al. and Comparing neural models using their perceptual discriminability predictions show that different internal geometries can produce the same behavior, and that similar geometries can produce different behavior. These papers treat the many-to-one mapping from representation to behavior as a general fact. None of them identifies a specific, measurable subspace through which a given model's behavior is actually computed, and that is what we supply.

The mechanism comes from the modality-gap literature. Mind the Gap shows that image and text embeddings sit in separate cones and that shifting embeddings along the gap vector changes zero-shot accuracy and fairness. Is the Modality Gap a Bug or a Feature? proves that the gap is orthogonal to the subspaces spanned by each modality, and it builds a post-processing step on that structure. Neither paper asks what the gap means for human alignment. We reuse their geometric decomposition and redirect it toward a question about cognitive alignment.

The Decomposition

We write $z$ for an image embedding and $t_1,\dots,t_C$ for the text embeddings of a concept vocabulary. We center the text embeddings across concepts, because their shared mean only adds the same offset to every class score and so cannot change which class wins. We define $P_k$ as the projector onto the top $k$ principal directions of the centered text embeddings. The image embedding then splits as

$$z = P_k z + (I - P_k) z,$$

where $P_k z$ is the readout component and $(I-P_k)z$ is the silent component, which includes the global gap direction. Zero-shot class scores are $s_c = \langle z, t_c\rangle / (\lVert z\rVert \lVert t_c\rVert)$. Image normalization does not change which class wins, so when $k$ covers the full span of the centered text matrix, the silent component has no effect on choices over that vocabulary. We pick $k$ as the smallest value that preserves 99% of zero-shot decisions. We then report all results as curves over $k$, so the conclusions do not depend on one threshold.

Invariance on the vocabulary used to build $P_k$ is true by construction. The empirical content lies elsewhere. First, we test whether a subspace built from one vocabulary also governs behavior on a different vocabulary. We build $P_k$ from the 1,854 THINGS concept names and evaluate on the 16 Model-vs-Human categories, and the reverse. If the readout subspace is a general property of the model and not an artifact of one label set, it should transfer. Second, we measure how much representational alignment is carried by each component.

Research Program

The first study is an audit. For CLIP ViT-L/14, OpenCLIP ViT-H-14, SigLIP, and a family of OpenCLIP checkpoints that differ in size and training data, we compute standard alignment measures on the full embedding, the readout component, and the silent component. The measures are linear-ridge encoding of Natural Scenes Dataset fMRI across early and higher ventral regions, odd-one-out triplet accuracy and similarity-matrix correlation on THINGS, and NIGHTS mid-level triplets. We extract activations with THINGSvision. The comparison that controls for dimensionality is a random subspace of the same size. If the silent component predicts higher visual cortex nearly as well as the full embedding, then a model's brain score is partly a score for structure its decisions never use.

The second study is a causal edit, in two directions. "Behavior-preserving representation edits" apply a human-alignment transform only to the silent component. We use the gLocal transform from Improving neural network representations using human similarity judgments, a learned linear map fit to THINGS triplets while preserving nearest neighbors, restricted to $I-P_k$. If it raises triplet and fMRI alignment while leaving zero-shot choices on Model-vs-Human and ImageNet unchanged, we have a linear analogue of the model metamers idea: representations that differ in human-likeness and yield identical behavior. "Representation-targeted behavior edits" apply the same transform only to $P_k z$, and we predict that these move zero-shot error consistency.

We also take known interventions and locate where they act. The first is test-time low-pass filtering from Low-Pass Filtering Improves Behavioral Alignment of Vision Models, which blurs input images and improves error consistency and shape bias. The second is the uniformity-based gap closing of Its Not a Modality Gap, which fine-tunes CLIP to spread embeddings evenly over the sphere. For each, we measure how much of the embedding shift falls in the readout versus the silent subspace, and how each component's alignment metrics change.

The third study tests prediction across models. For each model we compute full-space and readout-space versions of each representational metric. We then ask which version better rank-predicts Model-vs-Human error consistency and shape bias across the model population. This is the most direct test of whether readout restriction removes part of the dissociation reported in How Aligned are Different Alignment Metrics?.

A tension in the file constrains the design. Fahim et al. close the gap by forcing uniform spread over the hypersphere. The THINGS and context results (Context Sensitivity Improves Human-Machine Visual Alignment) show that human similarity is clustered and hierarchical. We therefore expect uniformity training to raise behavior-relevant accuracy while lowering readout-space similarity-matrix correlation with humans. We treat this as a falsifiable prediction, not an assumption.

Baselines

The naive reference point is the standard practice: full-embedding RSA and encoding scores used as-is, with zero-shot behavior evaluated separately. Every claim is measured against the rank correlation this practice achieves between representational and behavioral alignment. The only population-level figure in the file is the 0.198 average pairwise neural-behavioral correlation from How Aligned are Different Alignment Metrics?. Our model population is narrower, so we recompute the full-space correlation on exactly our models and our metrics, and readout-space metrics must beat that recomputed value. We also report the 0.198 figure for context only.

The strongest existing method for making a contrastive model behave like humans is test-time low-pass filtering. Low-Pass Filtering Improves Behavioral Alignment of Vision Models reports error consistency of 0.37 for OpenCLIP ViT-H-14 on Model-vs-Human, versus Imagen's 0.31, and shape bias rising from 0.60 to 0.96, at a cost of 3–6% out-of-distribution accuracy. We run it with the reported blur width of about 2.5 pixels and also sweep the width, on the same checkpoints and images. A readout-subspace edit counts as beating it if it matches 0.37 error consistency with a smaller out-of-distribution accuracy drop. It also counts if it equals blur on behavior while raising rather than leaving untouched NSD encoding in the readout component. Blur's effect on NSD has not been measured at all, so we measure it.

The strongest existing representational intervention is the full-space gLocal transform. For CLIP ViT-L/14 it is reported to raise Entity-13 5-shot accuracy from 65.3% to 71.9% and CIFAR-100 anomaly-detection AUROC from 91.4% to 97.2%. We refit it with the same THINGS triplets, regularization, and tuning budget as our restricted versions. For CLIP triplet accuracy on THINGS and for NSD encoding scores of these specific CLIP models, no reported result was found on Lacuna, so we compute both ourselves. The gap-manipulation baselines are the mean-shift of Mind the Gap and the orthogonal translation of Is the Modality Gap a Bug or a Feature?. The latter is reported to improve Gaussian-noise robustness by up to 10%, but its effect on human alignment has never been measured.

The key ablation replaces $P_k$ with a random subspace of the same dimension, and also with the top-$k$ principal directions of the image embeddings themselves. If either matches the text-derived subspace on the audit or the cross-model prediction, then the text readout is not the explanation, and our claim fails.

Why not just…?

Why not just correlate modality-gap size with alignment scores across models? Gap size varies with architecture, data, and scale, all of which independently affect alignment. Pasquiou et al. and Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens show that alignment can be non-monotonic in scale. A cross-model correlation would be confounded. Our edits act within a single fixed model and say which subspace causes which change.

Why not just measure behavior with linear probes instead of text prompts? A linear probe trained on the full image embedding builds a new readout and so no longer measures the model's own behavior. We include probe behavior as a secondary condition. We expect probes to recover structure from the silent component, and that would itself confirm that the component is informative but unused by the text readout.

Why not just blur the input, which already gives the best error consistency? This is a real threat for anyone who only cares about Model-vs-Human scores, and we do not claim to beat blur on error consistency alone. Blur is unimodal, it costs 3–6% out-of-distribution accuracy, and its effect on neural alignment is unknown. Our contribution is to say where its effect lands. If blur turns out to act almost entirely in the readout subspace and raises readout-space NSD encoding, that is a positive finding about blur, and our decomposition is what reveals it.

Isn't the decomposition tautological? On the vocabulary used to build $P_k$, behavioral invariance holds by construction, and we say so. Three things are not guaranteed: transfer across vocabularies and prompt templates, the share of fMRI and triplet alignment carried by the silent part, and readout-space metrics predicting behavior across models. Each can come out against us.

Decisive Checks

Three results would settle the proposal. In the first, the silent component of OpenCLIP ViT-H-14 retains a large share of its full-embedding NSD encoding in higher ventral regions, and a large share of its THINGS triplet accuracy, well above a random subspace of the same size. In the second, gLocal restricted to that component raises human alignment while zero-shot decisions on Model-vs-Human change for under 1% of trials, using a subspace built from THINGS vocabulary and not from Model-vs-Human labels. In the third, across the model population, readout-space metrics rank-predict error consistency and shape bias better than the recomputed full-space correlation, while image-PCA and random subspaces do not.

The opposite pattern falsifies the core claim. That pattern would be negligible alignment in the silent component, edits that fail to transfer across vocabularies, or no gain in cross-model prediction. In that case, the modality gap would have to be dropped as an explanation for the dissociation between representation and behavior.