Stereotype Fallback: Does Social Bias Grow When Multimodal Models Lose Track of Long Documents?

Vision-language models, which read images and text together, are known to hold stereotypes. They link "nurse" to women and "engineer" to men, and they judge competence from hairstyles. These models are also known to lose track of pictures buried in long inputs. This proposal argues that the two failures are linked. Take a person whose photo sits on page 2 of a report and whose actual role is stated on page 14. When the model can no longer connect the two, it does not guess at random. It falls back on what the person looks like, and so on stereotype. If that is true, bias in long multimodal documents grows with the distance between a person's image and the evidence about them. Short-context bias benchmarks cannot see this, and short-context debiasing will not fix it.

We propose to measure this "stereotype fallback" directly, then test a targeted remedy. The remedy is a hierarchical document encoder for vision-language models that compresses each page into separate per-person records rather than one pooled page summary. A hierarchical encoder reads a long document in pieces and then combines compact summaries of the pieces. The worry is that such compression keeps visually salient appearance cues and drops individuating facts. Our encoder is built so that those facts survive compression.

Core Claim

We make two claims, and the second depends on the first. The first is diagnostic. In long interleaved documents, a model's errors about people become increasingly stereotype-directed as the gap between a person's image and the facts about them grows. Accuracy on stereotype-congruent cases stays high, and accuracy on counter-stereotypical cases falls. The second is architectural. Hierarchical encoding either causes or cures this effect, depending on what the summary slots are forced to represent. Pooled page summaries of the kind used in text-only hierarchical attention should lose the fact that ties an entity to its evidence. Entity-anchored summaries, which keep "this face, this name, these stated facts" as one record, should flatten the growth of the bias gap with distance.

The evidence motivating the diagnostic claim comes from two literatures that have not met. On the grounding side, Needle In A Multimodal Haystack shows that image needles degrade much faster than text needles as context grows from 1k to 72k tokens. In other words, a model's effective visual context is far shorter than its nominal one. PRISMM-Bench shows that the same figure-text inconsistencies found in isolated snippets are missed in full papers. LongHalQA shows that long context turns correct visual descriptions into contradictions of the visual evidence. On the bias side, SB-Bench finds that adding an image pushes models away from "cannot be determined" and toward stereotypes. StylisticBias finds that about 15 salient appearance cues drive roughly 80% of social-judgment variance. Taken together, these results predict that when grounding to the individuating evidence decays, the salient appearance cues remain and fill the gap. None of these papers tests that prediction. Every bias benchmark in the file uses single images or single scenes, and every long-context benchmark measures factual accuracy only.

Relation to Prior Work

Three prior works come closest. Bias in Danish Medical Notes combines hierarchical long-document encoding (chunked BERT plus an upper Transformer) with subgroup error analysis. It finds persistent sex and cancer-type disparities in false positive rates that resampling only partly removes. That work motivates our concern that hierarchy does not by itself remove bias. However, it is text-only, it has no cross-modal linking, and it does not ask how disparity varies with the location of evidence. HiCI supplies our method template, which has three steps. Learnable query slots summarize each segment. The segment summaries are pooled into a global workspace using mean, variance, and extremes. That workspace is then prepended to every segment's attention memory. HiCI is text-only; its authors leave vision-language extension to future work, and it never evaluates bias. Our specific departure is to replace statistical pooling with entity-indexed records. Pooling is exactly the operation we expect to lose the individuating link. Document Haystacks introduces V-RAG, which combines a CLIP/SigLIP/OpenCLIP retrieval ensemble, a yes/no filter model, and a reader model over up to 1,000 document images. It is the strongest retrieval-style competitor. It treats each page as atomic and is scored only on factual retrieval.

On the mitigation side, FairReason debiases with reinforcement learning on a 1:4 debias-to-reasoning data mix. Revealing and Reducing Gender Biases in VLAs finds that low-rank adapter fine-tuning (LoRA) is the best bias-versus-capability trade-off. FairVLP adds a counterfactual KL penalty during pre-training. All three change the model's prior on short inputs. We ask whether a debiased prior produces correct long-context answers or merely more abstentions. Two papers constrain the design. SB-Bench's "thinking penalty" warns that chain-of-thought can rationalize stereotypes, so we cannot assume reasoning prompts help. Racial and Gender Stereotypes Encoded Into CLIP Representations warns that CLIP-family retrievers, which are the core of V-RAG, may themselves be stereotype-biased.

Research Program

The first component is a controlled benchmark of counter-stereotypical evidence at a distance. Each item is a synthetic multi-page document, such as a project report, a news feature, or a staff directory. In each document, a person appears in a photo with a caption or figure number, and a text passage elsewhere states that person's occupation, action, or trait. The passage refers to the person only through the figure number or name, never through appearance. The question shows the person again, or refers to "the person in Figure 3," and asks for the stated fact. Answering therefore requires linking an image to distant text.

Person images come from SB-Bench's non-synthetic, visually grounded images, which avoids the synthetic-face realism concern raised against StylisticBias and GAB. Occupations follow the gender-stereotyped sets used in the VLA audit. Filler pages come from DocHaystack and InfoHaystack document images and from PRISMM-Bench paper pages. We vary five factors: the token and page distance between photo and evidence, the number of other people pictured (following GABInsight's finding that a second-gender distractor sharply worsens binding), the position in the document, and whether the stated fact is stereotype-congruent or incongruent. Each congruent item has a matched incongruent twin with identical layout and filler; only the fact or the pictured person is swapped, in the spirit of CounterBias.

We score four quantities as functions of distance $d$. The first two are accuracy on congruent items, $A_c(d)$, and accuracy on incongruent items, $A_i(d)$. The third is the stereotype gap $G(d) = A_c(d) - A_i(d)$. The fourth is the stereotype-substitution rate: the fraction of errors whose answer is the stereotyped option rather than another distractor. Answers use a structured output format, as in PRISMM-Bench, so that multiple-choice style cues cannot be exploited. A text-only "blind" condition, with the image removed, estimates how much the prior alone contributes.

The second component is an entity-anchored hierarchical encoder built on an open vision-language model such as Qwen2-VL or InternVL-1.5, with the backbone frozen and HiCI-style adapters trained at a similar parameter budget. For each page, a detector proposes person regions. Local construction then uses cross-attention slots that are each tied to one region together with its caption and its co-referring text spans, producing one record per entity. Global integration keeps these records separate and indexed rather than averaging them. HiCI's pooled page summaries are kept alongside the records. The records are prepended to the attention memory of the question segment. We train on generated documents disjoint from the benchmark templates, with a balanced congruent/incongruent mix, plus an auxiliary loss that asks the model to point to which record supports its answer.

Baselines

The naive reference point is the unmodified base model reading the whole document as one flat sequence. It sets the $G(d)$ curve that everything else must improve on. The same model on the focused condition, with only the photo and the evidence sentence, gives the no-distance ceiling, which lets us separate a distance effect from a baseline bias.

The strongest existing long-context method is V-RAG from Document Haystacks. It is the strongest competitor because it removes distance altogether when retrieval succeeds. We run it with its published encoder ensemble, filter, and top-5 pages, using the same backbone as the reader. The file reports V-RAG's gains only in relative terms: 9% on DocHaystack-1000 and 11% on InfoHaystack-1000 in Recall@1 over prior baselines. On MM-NIAH, retrieval raises InternVL-1.5 from 42.4% to 50.1% overall. There is no reported result found on Lacuna for V-RAG or any retrieval method on a bias metric. To beat V-RAG, our method must achieve lower $G(d)$ at the largest distance without lower $A_c$, and in particular higher $A_i$ at the largest distance. We report retrieval recall separately for congruent and incongruent items to show where each method fails.

The strongest bias-mitigation baseline is LoRA debiasing following the VLA audit, together with FairReason-style reinforcement learning at the 1:4 mix. FairReason reports a 10% stereotype-score reduction at 88% reasoning retention. We apply both to the same backbone with the same adapter parameter count and the same number of optimizer steps as our encoder.

The ablations isolate the key ingredient. The first is HiCI-style pooled slots with no entity records, adapted to vision-language inputs, using the same parameters and the same training documents. The second is the flat model fine-tuned on our training documents, which separates the architecture's contribution from the counter-stereotypical data. The third is our encoder without the pointer loss. The claim holds only if entity records beat pooled slots and flat fine-tuning at matched data. Otherwise, the gain is just the training data.

Why not just…?

Why not just debias the model on short inputs and let it carry over? This is a real threat to the method claim, though not to the diagnostic one. A successfully debiased prior could turn stereotype-directed errors into abstentions, which would shrink $G(d)$. It would do so without getting counter-stereotypical people right. Our metrics tell these outcomes apart: fairness through grounding raises $A_i(d)$, while fairness through abstention lowers $A_c(d)$ or raises the "cannot determine" rate. The VLA audit already warns that heavier debiasing degrades general capability. If LoRA or GRPO debiasing matches our encoder on $A_i$ at large distance, we will conclude that the prior, not the architecture, is the lever, and say so.

Why not just retrieve the relevant pages with V-RAG? If retrieval were perfect, distance would not matter. But the question here asks about a pictured person, so retrieval must match on appearance, and CLIP-family encoders carry race and gender stereotypes. We therefore expect retrieval to surface stereotype-congruent pages more reliably, which would move the bias into the retriever rather than remove it. Per-condition recall will show directly whether this happens. If V-RAG recall is equal across congruent and incongruent items and $G(d)$ stays flat, retrieval suffices, and the encoder is unnecessary for this task.

Why not just caption each page and reason over text? A caption-then-reason pipeline is itself a crude hierarchical encoder. It might look fair because captions drop appearance. But our items require matching the question's image to a figure, so appearance must be kept in some form. Captions that describe people will reintroduce exactly the salient cues StylisticBias identifies. We include this pipeline as an extra baseline and report whether its captions mention demographic attributes.

Why not just prompt the model to reason carefully or "be unbiased"? SB-Bench finds that long reasoning traces rationalize stereotypes, and the VLA audit finds prompt engineering the weakest mitigation. We include chain-of-thought and "be unbiased" prompts, but expect them to leave $G(d)$ unchanged or larger.

Decisive Checks

The first check decides whether the program continues. For the flat base models, $G(d)$ must rise with distance while $A_c(d)$ stays near its focused-condition value, and the stereotype-substitution rate must exceed the rate expected if errors were spread evenly across distractors. If errors at long range are not stereotype-directed, the phenomenon is ordinary long-context forgetting. In that case we report the negative result and drop the bias-specific framing.

The second check is the matched ablation. Entity-anchored records must reduce the slope of $G(d)$ relative to both pooled HiCI slots and flat fine-tuning trained on identical documents. A gain that disappears under matched data would mean that counter-stereotypical training data, not hierarchy, did the work.

The third check is that the method does no harm elsewhere. We will report MM-NIAH, where Gemini-1.5-Flash leads at 55.3% overall; DocHaystack; PRISMM-Bench at the document level, where GPT-5 reaches 53.9% and humans 65%; and SB-Bench fairness, where Gemini-2.5-Flash-Lite leads at 80.89%. Our encoder should not fall below the base model's own scores on these. If it does, the bias reduction is coming at the cost of general grounding.