Small open models now match much larger ones on knowledge-heavy visual question answering when the retrieved knowledge is compressed before it reaches them. Instead of raw Wikipedia passages, the model receives compact entity-relation triplets such as (Subject, Relation, Object). We think this compression has a hidden cost. A triplet keeps the fact and drops the hedges, qualifiers, and surrounding counter-evidence. What remains looks a lot like the canonical patterns the model memorized in pretraining, so the model may stop reading the evidence and answer from habit. Benchmarks reward this whenever habit and evidence agree, so it stays invisible. It shows up only when they disagree: on adversarial splits that keep a question's wording but change its answer, and on cases where the retrieved fact contradicts what the model already believes.
This note proposes two things. The first is a controlled test of whether more compression makes a 7B reader more shortcut-prone and more likely to override evidence. The second is a repair we call evidence-bound answering. The answer is written into a small template, and decoding forces each slot either to copy an entity that appears in the retrieved context or to name the triplet that supports it. A third outcome is always allowed: an explicit "insufficient evidence" slot. Put simply, we ask whether the compression that makes small models good also makes them easy to fool. We then test whether forcing the answer to point at the evidence fixes this without giving back the gains.
Core Claim
The central hypothesis is that aggressive structural compression of retrieved knowledge improves clean accuracy partly by amplifying the reader's reliance on memorized associations. If so, compressed-context systems should lose more than flat-passage systems on two kinds of test. One is shortcut-exposing adversarial splits. The other is counterfactually edited evidence. Both use the same retriever and the same reader. The second hypothesis is that adding a format template alone will not close this gap. A template whose slots are constrained to retrieved entities, and that carries an explicit provenance slot, should close a substantial part of it. That template must also give up no more than about one point of clean accuracy. It must handle retrieval failure by abstaining rather than by confidently copying a wrong entity.
The motivation comes from three findings in the file. The compression system Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning (SCoRe) distills retrieved passages and visual notes into triplets for Mistral-7B and reaches 52.1% on OK-VQA and 52.4% on A-OKVQA zero-shot. Its authors also report that when retrieved triplets contradict the model's pretrained beliefs, the model falls back on those beliefs. FVQA 2.0 shows that fact-based VQA models score about 88% when the answer is fixed and the question rephrased, but drop to about 69% when the question wording is fixed and the image and answer change. That gap is the signature of memorized question-to-answer shortcuts. Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? documents the same surface-sensitivity in pure reasoning. No paper in the file runs a compressed-context reader on these adversarial or counterfactual conditions. That is the gap we target.
A further finding constrains the design. Template Filling for Controllable Commonsense Reasoning shows that slotted templates improve factual consistency (FactCC 79.88% versus 65.27%). Even so, 22% of its errors are factually incorrect and 53% are generic, because the slots are filled from parametric memory. Templates without grounding control form, not truth. Our template therefore has to bind slot contents to retrieved evidence, not just impose structure.
Research Program
The first component is a compression ladder. A shared retriever is held fixed: SCoRe-style two-stage title-to-paragraph retrieval for OK-VQA and A-OKVQA, and an EntRAG-style entity-centric retriever for INFOSEEK. Its output feeds a fixed reader, Mistral-7B, chosen to match SCoRe. The ladder has four rungs. The first is raw retrieved passages. The second is pruned passages in the spirit of TCRA-LLM. The third is SCoRe's relational logic distillation into plain triplets. The fourth is "qualified triplets," which keep negation, numeric bounds, and time qualifiers as extra fields. SCoRe's plain triplets cannot express these, and the qualified rung tests whether the loss comes from compression itself or from discarding qualifiers. Each rung's token count is recorded so that results can be read against cost.
The second component is the evaluation conditions. Each benchmark is evaluated in its clean form and under two stressors. The first is shortcut exposure, using FVQA 2.0's FixQ and FixA splits. The second is counterfactual evidence editing. In the retrieved context, we replace the object of the answer-bearing triplet, together with every passage mention of it, with a type-matched alternative from the same knowledge source. We then score whether the answer follows the edited evidence or the original prior. Editing passages and triplets consistently keeps the ladder comparison fair. A third condition separates retrieval failure from reading failure. On INFOSEEK we run the reader with predicted and with oracle entities. The source paper Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? shows this switch moves CLIP+FiD from 18.2% to 45.6%.
The third component is the repair, evidence-bound answering. The reader emits a short template: a supporting-triplet slot filled with an identifier of a context triplet, an answer slot, and an abstain option. Decoding is restricted with a trie over the retrieved context. The answer slot can emit only strings that appear as objects or subjects of context triplets, or as spans of passages under the raw-passage rung. The provenance slot can emit only valid triplet identifiers. Open-vocabulary answers that require light normalization, such as plurals or units, are handled by allowing the answer slot to emit a normalized form of an evidence span. The evidence-bound reader is used zero-shot first. If zero-shot compliance is poor, it is then trained on the training split with a small reinforcement-learning stage in the style of ContextIF and StepTool. The rewards in that stage are step-level: one for citing a triplet that actually entails the answer, one for correct abstention when no context triplet contains the gold answer, and one for final correctness. Unlike ContextIF, the structured block points outward at retrieved evidence rather than at self-generated demonstrations. Unlike MMhops-R1, the intermediate text is constrained instead of free-form.
Two prior designs are related and differ in specific ways. GNN-RAG supplies candidate answers from a graph network as verbalized paths, but lets the LLM answer freely. Our design makes the binding to evidence a decoding constraint and asks whether that constraint changes behaviour under conflict. Visual Chain-of-Thought Prompting verifies free-text rationales against pixels with CLIP. We verify against retrieved symbolic knowledge instead, and we put the check inside decoding rather than after generation.
Baselines
The naive reference point is the same Mistral-7B reader given only the image caption and question, with no retrieved knowledge. This reference measures how much any system actually uses external knowledge. On the counterfactual condition it serves as a pure "prior-following" baseline. A system whose answers under edited evidence resemble this baseline's answers is ignoring the evidence.
The flat-retrieval baseline gives the same reader raw retrieved passages: the top rung of the ladder, with the same retriever and passage count. It is the strongest competitor on the robustness question in particular. If compression really strips counter-evidence, flat passages should be more robust, and the repair would have to beat flat passages on robustness while keeping compressed-context accuracy.
The strongest existing method on the clean benchmarks is SCoRe, which reports 52.1% on OK-VQA and 52.4% on A-OKVQA zero-shot with Mistral-7B, and 55.7% and 54.9% in 16-shot (Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning). We will reimplement it with its retriever and reader and confirm we reproduce within about one point before adding anything. Beating it means two things together. On clean data, staying within one point of its accuracy. On FixQ-style and counterfactual conditions, a gain we pre-register as at least five points of evidence-following rate.
On FVQA 2.0 the strongest reported robust system is RAVQA-DPR trained with adversarial augmentation, at 85.05% (FVQA 2.0). On INFOSEEK the strongest reported number in the file is EntRAG at 44.5 (EntRAG). There, our target is not to exceed EntRAG's accuracy. It is to show that evidence-bound answering on top of an EntRAG-style retriever converts retrieval failures into abstentions rather than wrong answers. For a frontier proprietary model on these exact settings, we found no reported result on Lacuna for GPT-4o on OK-VQA, A-OKVQA, or FVQA 2.0. We will run one as an untuned reference rather than as a tuned competitor.
Three ablations isolate the key ingredient. The format-only ablation keeps the template tags but removes the decoding constraint, in the spirit of TORSO. It tests whether structure alone helps. The constraint-only ablation restricts the answer slot to context entities but drops the provenance slot. The no-abstain ablation removes the abstain option, to measure how often forced copying produces confident wrong answers when retrieval misses. Every variant shares the retriever, the reader weights, the number of retrieved items, and any training budget, measured in queries and update steps. Any trained variant is compared against a flat-passage reader given the identical training stage.
Why not just…?
Why not just tell the model to answer only from the provided context? This is the cheapest repair and a real threat. SCoRe's own observation that the model reverts to its beliefs under conflict suggests that instructions alone are not enough at 7B. Still, that is one observation, not a measurement. We include an instruction-only variant at every ladder rung. If it matches evidence-bound decoding on counterfactual evidence-following, the constraint is unnecessary and we will report that.
Why not just keep raw passages and skip compression? If raw passages turn out to be both robust and accurate, compression has no case beyond cost. The file's evidence cuts the other way on accuracy and cost. SCoRe beats its flat-prompt predecessor and cuts latency from about 29.5s to about 17.5s per sample, and Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models reports that unfiltered long context brings in confounders. The ladder answers this directly by reporting accuracy, robustness, and tokens for each rung. The qualified-triplet rung tests whether a middle point dominates both ends.
Why not just train on adversarial or counterfactual data? FVQA 2.0's augmentation lifts ZS-F-VQA by 24.09 points and shrinks the clean-to-adversarial gap from over 10 to about 3 points. MUTANT gains 10.57 points on VQA-CP v2 using mutated pairs and consistency losses. This is a strong alternative wherever matching adversarial data exists. Its limitation is that it teaches robustness to the perturbation families it saw. We test that limitation in two steps. We train a flat and a compressed reader on FVQA 2.0 augmentation, then evaluate them on our counterfactual triplet edits for OK-VQA and A-OKVQA, which they never saw. If augmentation transfers as well as the decoding constraint, we will recommend augmentation as the simpler route.
Why not just use a bigger model? In the file, scale alone fails on long-tail knowledge: PaLI-X scores 10.8% on INFOSEEK-Human. A structured-context 7B model matches GPT-4 on CWQ (62.8% versus 62.6% Hit, per GNN-RAG). Larger models may hold stronger priors and so override evidence even more. The untuned frontier reference run on our counterfactual condition will show which way this goes.
Proposed Experiments
The first decisive check tests the premise. At equal retrieval, does moving from raw passages to plain triplets lower FVQA 2.0 FixQ accuracy and lower the counterfactual evidence-following rate on OK-VQA and A-OKVQA? If not, the "compression invites shortcuts" claim is false. In that case the paper becomes a clean negative result showing compressed context is safe, which is itself worth publishing given how widely compression is used. The second check tests the repair. Does evidence-bound answering on the triplet rung recover robustness to at least flat-passage level while staying within one point of reproduced SCoRe on clean accuracy? The format-only and instruction-only ablations show whether the constraint, rather than the tags or the wording, is responsible. The third check targets the retrieval-failure regime on INFOSEEK. Comparing predicted against oracle entities, we measure wrong-answer rate and abstention precision. A useful repair should leave oracle-entity accuracy unchanged and shift predicted-entity errors toward abstention. If forcing answers to copy from the context instead raises confident errors, the no-abstain ablation will show it. As a final sanity check on text-only multi-hop knowledge, we apply the same counterfactual edits to GNN-RAG's verbalized paths on CWQ, where the reported reference is 62.8% Hit. This tests whether the effect extends beyond vision. Metrics throughout are benchmark accuracy (Hit for CWQ), evidence-following and prior-following rates under edits, abstention precision and recall, and context tokens per query.
References
- Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning
- FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- Template Filling for Controllable Commonsense Reasoning
- TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- ContextIF: Enhancing Instruction-Following through Context Reward
- StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning
- MMhops-R1: Multimodal Multi-hop Reasoning
- GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning
- Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- TORSO: Template-Oriented Reasoning Towards General Tasks
- Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
- MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering