People from different cultures feel different things when they look at the same painting, and they disagree in patterned ways. Multimodal large language models, which are vision encoders attached to a text model through a small learned "projector," do poorly exactly where that disagreement matters. On the MVEI benchmark, GPT-4o reaches 84.3% on explaining an image's emotion but only 69.2% on judging how different viewer personas would react, against 91.6% for humans overall (Customizing Visual Emotion Evaluation for MLLMs). This proposal asks where, inside the model, the difference between viewers disappears. It then tests whether moving the persona signal earlier, into the projector itself, brings that difference back.
The measuring tool is representational similarity analysis (RSA). RSA compares two systems without matching their coordinates. For each system, you build a matrix of how dissimilar every pair of stimuli looks, then correlate the matrices. Human annotations from separate cultural groups give one such "disagreement geometry" per group. We ask whether a model told to take a given culture's perspective reproduces that culture's geometry better than the other cultures' geometries. We borrow one idea from cryptography's adaptive security gap: a guarantee proven for a target fixed in advance says little about a target the adversary picks after seeing the system. We apply this to evaluation. Every alignment score is reported twice: on a fixed set of stimuli, and on a set chosen adversarially after the model is trained.
Core Claim
Our hypothesis is that persona prompting fails for a structural reason, not a data reason. In current models the projector maps an image to the same visual tokens whatever the persona. Any viewer-specific reinterpretation must therefore be built later, by the language model attending from image tokens to persona text. We expect a layer-by-layer RSA to show that persona-specific geometry at image-token positions stays near zero through the early and middle layers, and appears weakly, if at all, only near the output. If so, we can make a testable prediction. A projector that receives the persona as input, trained to match each group's human dissimilarity structure, should raise culture-specific alignment beyond what equally tuned language-model adapters achieve. It should do so on the fixed stimulus set and also on adversarially chosen ones.
The quantity that matters is not plain alignment with humans. Plain alignment is dominated by structure every culture shares, such as "funerals look sad." We instead measure cultural contrast. Let $H_g$ be the human dissimilarity matrix for group $g$. Let $M_g$ be the model's dissimilarity matrix when conditioned on persona $g$. Let $\rho$ be Spearman correlation over the upper triangle. Then
$$ \Delta = \frac{1}{G}\sum_{g=1}^{G}\Big[\rho(M_g, H_g) - \frac{1}{G-1}\sum_{g' \neq g}\rho(M_g, H_{g'})\Big], $$
where $G$ is the number of groups. $\Delta$ is zero when the model gives the same geometry regardless of persona, even if that geometry matches humans well. It is positive only when conditioning on a culture moves the model toward that culture specifically.
Research Program
Data and human geometry. The main testbed is ArtELingo (ArtELingo), which has over 1.2 million emotion annotations of 80,000 WikiArt images. The annotators are native English, Arabic and Chinese speakers, and they diverge on nudity, alcohol and religious imagery. For each language group and image we form the distribution over the 9 emotion categories. The dissimilarity between two images is the Jensen–Shannon divergence between their distributions, giving $H_{\text{en}}, H_{\text{ar}}, H_{\text{zh}}$. We estimate a noise ceiling for each matrix by splitting each group's annotators into halves. We also compute the ceiling for $\Delta$ itself: human half A of group $g$ scored against half B of $g$ versus half B of the other groups. This matters because some apparent cultural divergence may be annotator noise.
Diagnosis. We use two open models evaluated on MVEI, InternVL2.5 and Qwen2.5-VL, and hold the image fixed. We vary only a persona instruction ("You are a native Arabic-speaking viewer…"). At every language-model layer we extract hidden states at the image-token positions and at the answer position. We build $M_g$ per layer with correlation distance and compute $\Delta$ as a function of depth. The same analysis applies to the model's generated emotion distribution, giving a behavioral $\Delta$. This parallels the text-only comparison of hidden-state and behavioral geometry in From Associations to Activations, which never touched visual tokens or viewer-dependent judgments. MVEI treats the model as a black box. This step supplies the missing "where." Because RSA is known to depend on the distance metric, we repeat the analysis with Euclidean distance and report whether conclusions change.
Intervention. We replace the deterministic projector with a persona-conditioned one. The persona embedding (culture identity, or a free-text persona encoded by the frozen language model) scales and shifts the projector's hidden units, a feature-wise modulation. Vision encoder and language model stay frozen, following the staged-training finding that freezing both protects their pretrained representations (Scalable Vision Language Model Training). The training loss combines two terms. The first is a KL term on each group's emotion distribution, the distribution-learning recipe of Seeking Subjectivity in Visual Emotion Distribution Learning. The second is a differentiable RSA term: one minus the Pearson correlation between minibatch model and human dissimilarity matrices, computed at a middle language-model layer. The RSA term matters because a pure KL loss can be satisfied by per-image output adjustments. The RSA term asks the model to rearrange the relations between images the way a group does. Conditioning the projector differs from Video-LLaVA and Deep Pre-Alignment for VLMs, which align visual features to text more closely but leave the mapping identical for every viewer. It differs from SAMNet, whose multiple "virtual annotator" branches live in a vision-only network and never reach a language model.
Adaptive evaluation. An RSA training loss invites a failure: the model can learn to match the anchor images it sees while the geometry on other images stays wrong. In the cryptographic setting, selective security fixes the target before setup. Moving to adaptive security, where the adversary picks after seeing public parameters, costs a guessing loss that can be exponential (Efficient Selective IBE Without Random Oracles). Gandalf showed the same failure empirically for language-model defenses: static tests overstate robustness against attackers who adapt (Gandalf the Red). Our analogue works as follows. After training, an evaluator sees the model. It greedily selects $K$ images from a held-out pool of already-annotated ArtELingo images to minimize $\Delta$, so no new human labels are needed. We define the adaptive gap as
$$ \text{gap} = \Delta_{\text{fixed}}(K) - \Delta_{\text{adaptive}}(K), $$
where $\Delta_{\text{fixed}}$ is the average over random $K$-subsets. A search over subsets will always find some low-scoring set, so we run the identical search on the human split-half comparison. Only the model's gap in excess of this human gap counts as fragility. An optional stress test reuses boundary-image synthesis (Synthesizing Images on Perceptual Boundaries of ANNs). It generates images where the persona-conditioned models disagree most, and collects a small set of new ArtELingo-style annotations on them.
Baselines
The naive reference point is the unmodified model with no persona instruction, scored against each group's matrix. By construction its $\Delta$ is zero. It tells us how much shared structure exists before any conditioning.
The second reference is the unmodified model with persona prompts. This is the practice MVEI already tests and finds weak (Perception Subjectivity: 69.2% for GPT-4o, 61.3% for InternVL2.5, per Customizing Visual Emotion Evaluation for MLLMs). We report its $\Delta$ and MVEI Perception Subjectivity for both open models.
The strongest competitor is LoRA adapters on the language model, trained with the same KL targets per culture and prompted with the same persona. It is the cheapest way to let the model condition on culture, and it acts where persona information already enters. The reported LoRA gain on MVEI Perception Subjectivity is +4.0% for Qwen2.5-VL, against +15.4% on Sentiment Polarity. That LoRA was trained on binary statement judgments, not per-culture distributions, so a distribution-trained LoRA is a stronger and fairer opponent than the published number suggests. We match trainable parameter count to the conditioned projector, and use the same ArtELingo training split, epochs and hyperparameter search budget. Beating it means higher $\Delta$ on both fixed and adaptive anchors with overlapping-free bootstrap intervals over images, while per-culture KL on held-out images is not worse.
Two task-specific systems give external reference numbers. XLM-ACE reaches 72.4% overall accuracy on the combined ArtELingo test sets (ArtELingo). SAMNet reaches 74% accuracy with KL 0.40 on Flickr_LDL and KL 0.43 on Twitter_LDL (Seeking Subjectivity). We report our models' accuracy and KL on these same splits for comparability. No reported result was found on Lacuna for per-culture RDM alignment or $\Delta$ on ArtELingo, so all $\Delta$ numbers will be new, including those for the baselines.
The key ablations isolate the two ingredients. The first keeps the conditioned projector and trains with KL only, which tests whether the RSA term matters. The second applies an identical KL+RSA loss through a persona-conditioned adapter on middle language-model layers instead of the projector, which tests whether conditioning before the language model matters.
Why not just…?
Why not just prompt with personas? MVEI already tried this with detailed persona prompts, and Perception Subjectivity remained every model's worst dimension. Our diagnosis gives the mechanistic reason. If layer-wise $\Delta$ at image tokens is near zero, the persona text is not reshaping how the image is represented. If the diagnosis instead shows substantial $\Delta$ in middle layers that is lost only at the output, the problem is decoding, not projection. In that case we would redirect the intervention accordingly.
Why not just fine-tune the language model with LoRA on per-culture distributions? This is the real threat. The published +4.0% does not rule it out, because it used a different training target. We make it the main baseline under matched budget. If distribution-trained LoRA matches the conditioned projector on both fixed and adaptive $\Delta$, the claim that conditioning must happen in the projector is wrong. We would then report that culture-specific appraisal can live in the language model, which is still a useful finding.
Why not just train culture-specific classification heads like XLM-ACE, or a vision-only distribution learner like SAMNet? These heads predict per-image distributions, and on their own benchmarks they are strong. They cannot answer persona-conditioned questions in language, which is what MVEI measures. They also offer no internal geometry to probe. They remain in the comparison on accuracy and KL. If our model's KL is clearly worse than SAMNet's 0.40 on Flickr_LDL, then projection-level conditioning buys interpretability at a real accuracy cost, and we will say so.
Why not just use linear probes or a Procrustes fit instead of RSA? A probe asks whether culture-specific labels can be decoded from a layer. It cannot tell whether the model's relations between images reorganize per culture. $\Delta$ answers the second question, which is the one subjectivity raises. We still report linear-probe accuracy for culture-specific distributions at each layer. If probes and $\Delta$ disagree about where divergence appears, that disagreement is itself informative about the RSA critique that correlated geometry need not imply shared computation.
Decisive Checks
Three results settle the proposal.
First, the diagnosis. Layer-wise $\Delta$ for persona-prompted InternVL2.5 and Qwen2.5-VL on ArtELingo should stay below 20% of the human $\Delta$ ceiling at image-token positions through the middle layers. If it does not, the core premise is false.
Second, the intervention. The conditioned projector trained with KL+RSA should exceed matched-budget distribution-trained LoRA in $\Delta$ on held-out images. It should also improve MVEI Perception Subjectivity over the persona-prompted base model by more than the +4.0% reported for LoRA. The KL-only and language-model-conditioning ablations should each fall between the two.
Third, robustness. The model's adaptive gap must not exceed the human split-half adaptive gap by more than the bootstrap interval. A model that wins on fixed anchors but loses under adaptive selection has learned to fit the anchors, not the cultures. We would report it as the RSA analogue of a selectively secure scheme failing adaptive attack.
References
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
- ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture
- From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs
- Scalable Vision Language Model Training via High Quality Data Curation
- Seeking Subjectivity in Visual Emotion Distribution Learning
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- Deep Pre-Alignment for VLMs
- Efficient Selective Identity-Based Encryption Without Random Oracles
- Gandalf the Red: Adaptive Security for LLMs
- Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressions