Multimodal models that answer questions about charts and diagrams are fragile. Delete a chart's title, add grid lines, or reword the question without changing its meaning, and the answer often changes. Many deployed systems first look up a few solved "demonstration" examples similar to the incoming question and paste them into the prompt. This note proposes to test a simple, unexamined hypothesis: a large share of that fragility enters through the lookup step. A meaning-preserving edit moves the query's vector, the search returns different demonstrations, and the model reasons from different examples. The edit changes the answer indirectly, by changing what the model was shown.
We will measure this directly. For each question and each perturbed variant, we swap demonstrations between variants and see whether the answer follows the demonstrations or the query. We then split retrieval instability into two sources. The first is the embedding moving. The second is the approximate search index amplifying that movement. An approximate nearest neighbor (ANN) index is a data structure that finds close vectors quickly by walking a graph of neighbors instead of scanning everything, and it can make small query shifts look large. If retrieval turns out to be a major cause, stability becomes something to engineer at the index, which is cheaper than retraining the reasoner. If it turns out to be a minor cause, that is also worth knowing, because the retrieval-for-reasoning literature currently assumes retrieval helps without checking whether it destabilizes.
Core Claim
In retrieval-augmented chart and math reasoning, we expect three things to hold, and each can be falsified. First, a measurable fraction of the answer changes caused by semantics-preserving perturbations are mediated by changes in the retrieved demonstrations. Second, graph-based ANN indexes add instability beyond what exact search over the same embeddings shows, because perturbed queries enter and traverse the graph differently. Third, keying the index on a perturbation-invariant representation of the query recovers most of the lost stability at one search per query. Candidate keys are a canonical text rendering of the question, or the set of embeddings of several cheap rewrites of it. We treat the first two claims as the main contribution. The third is a comparison of fixes that we expect a simple voting baseline to contest seriously.
Where This Sits
The evidence that perturbations break multimodal reasoning is strong. Plot Twist perturbs chart titles, grid lines and marker styles. It shows accuracy falling from as high as 79% to the 30% range when an answer-bearing title is removed, and PaLI-3 dropping from 29.5% to 21.3% when grid lines are added. It only diagnoses the problem, it studies models without retrieval, and it never asks where the perturbed inputs land in embedding space. That last question is where our work starts.
Separately, two papers make retrieval part of multimodal reasoning, and they disagree about how. DCoT retrieves three demonstrations by cosine similarity from an 80-example bank. It found that text-only query embeddings beat joint image-text embeddings because visual features added noise. CIRCLES argues that plain similarity retrieval falls into a "similarity trap" of shared but non-causal features. It adds attribute-edited counterfactual examples, raising Gemma3-4B on CUB-200-2011 from 65.40% to 71.97%. Neither paper tests whether its retrieved set survives a rewording of the query. That stability property is what we measure, and the two methods serve as retrieval policies whose stability we compare.
The closest methodological ancestor is CRCA-Context. It generates five deterministic, meaning-preserving views of GitHub issues: compressed, bulletized, noisy-user, nuisance-instruction and distractor-context. It shows that BM25's robust Hit@5 is only 0.560. It then proposes Stable Context Selection, which retrieves for every view and keeps files that appear consistently. We borrow its view operators and its robust-hit metric, and its consensus method is our strongest stabilization baseline. It differs from our setting in three ways: it studies only lexical retrievers on text-only code tasks, and it stops at retrieval quality without tracing effects into downstream answers.
On the index side, GATE shows that graph indexes such as HNSW navigate poorly when queries come from a different distribution than the indexed items. HNSW (Hierarchical Navigable Small World) is a multi-layer neighbor graph searched greedily from coarse to fine. GATE fixes the problem with learned entry-point selection and reports only a 1.2% in-distribution versus out-of-distribution gap. Its shifts are between modalities, text queries against image items. We ask the untested version of the same question: whether small, meaning-preserving shifts within one modality also change the routes the graph takes. For fixes that represent a query as a set of vectors, we reuse Chamfer-DiskANN, which indexes set-to-set Chamfer similarity (for each vector in one set, take its best match in the other) with sublinear search.
Research Program
The first step is a large demonstration bank. The instability we study only matters when search must be approximate, and an 80-item bank like DCoT's can be scanned exactly. We will take ChartQA's human- and machine-written training questions from ChartQA and add template-generated chart questions with executable rationales, following the tools-and-templates generator of Li et al.. This scales the bank to roughly a million items with worked solutions. We will embed each item three ways: text-only (the DCoT choice), joint image-text, and a canonical key. The canonical key is the question rewritten by a fixed normalizer into a terse imperative form, with nuisance clauses removed, optionally concatenated with an extracted data-table summary. The three embeddings will be indexed with exact search, HNSW, and GATE-style entry selection.
The second step is a perturbation suite applied to test queries. On the text side we use CRCA-Context's five view operators, adapted to chart questions, plus question-to-statement conversion as in Li et al.. On the visual side we use Plot Twist's operators: remove the title, replace it with a neutral or misleading one, add grid lines, change marker styles, and switch the rendering engine. The visual edits are applied to Chart101's synthetic charts, which can be re-rendered, and to ChartQA charts where the underlying data allow re-rendering. Every perturbation is checked by a human sample for preserved meaning, so we do not inherit the evaluator bias the file flags for NLI-judged equivalence.
The third step is the decomposition. For a query $q$, its perturbed variant $q'$, and a retriever $R$ returning $K$ demonstrations, define retrieval instability as $I(R) = 1 - |R(q) \cap R(q')| / K$, averaged over pairs. Index-induced instability is the excess of approximate over exact search on identical embeddings:
$$\Delta_{\text{index}} = I(\text{ANN}) - I(\text{exact}),$$
measured at matched recall of the true neighbors of $q$. Matching recall matters: without it, a weaker index would look unstable simply because it is less accurate. For HNSW we also log the entry point and the first few nodes visited, to check whether perturbed queries enter the graph in different regions, as GATE's analysis would predict.
The fourth step is the causal attribution, which is the central experiment. For each pair whose answer changed, we run the perturbed query with the original query's demonstrations, and the original query with the perturbed query's demonstrations. The fraction of changes that follow the demonstrations estimates how much brittleness retrieval mediates. We run this with open multimodal models of different sizes, and with GPT-4o if budget allows, since MATH-Vision reports GPT-4o as the strongest system at 30.39%.
The fifth step is the comparison of fixes. We test three: canonical-key retrieval; view-set retrieval, where the query and each bank item are stored as the set of embeddings of their cheap rewrites and searched with Chamfer-DiskANN; and CRCA-style per-view retrieval followed by voting.
Baselines
The naive reference point is zero-shot prompting with no demonstrations. It shows whether retrieval helps at all on these benchmarks and whether adding it makes the model more or less stable under perturbation. The standard pipeline is plain HNSW top-3 retrieval over joint image-text embeddings, in the style of RICES, with $K=3$ following DCoT's reported optimum. Everything else is measured relative to this pipeline.
The strongest existing retrieval policies are DCoT's text-only retrieval and CIRCLES. DCoT is strongest on accuracy with an actual reported gain: 2.47% on MM-Bench, 3.0% on ScienceQA, and 38.4% versus 36.1% on MM-Vet for LLaVA-1.5-13B. CIRCLES is strongest on causal grounding, at 71.97% versus 65.40% for RICES on CUB-200-2011 with Gemma3-4B. We found no reported result on Lacuna for either method on ChartQA or Chart101, so both will be rerun on our bank with the same models, $K$ and token budget.
The strongest stabilization baseline is CRCA-Context's Stable Context Selection, run with the same view operators, the same number of views as our view-set method, and the same embedding model. Its only reported numbers are for lexical retrieval: BM25 robust Hit@5 of 0.560, and F1@5 falling from 0.203 to 0.187. We found no reported result on Lacuna for dense or graph-index retrieval, so the baseline number will come from our rerun. On the index side, GATE's entry selection is the strongest competitor for reducing $\Delta_{\text{index}}$. It will be trained on the same bank with its published hub and query features.
To beat these baselines, a fix must meet three conditions together. It must reduce answer changes on perturbed pairs by a margin outside seed and bootstrap variation. It must not lose clean accuracy to DCoT or CIRCLES. And it must reach Plot Twist's collective accuracy, meaning every elementary question about one chart answered correctly, at no more than one index traversal per query. Two ablations isolate what drives any gain. The first swaps canonical keys for raw text keys inside the same index, which isolates the key. The second runs exact search with the same keys, which isolates the index.
Why not just…?
Why not just use exact search, since demonstration banks are small? For banks of DCoT's size, that is the right answer, and our $\Delta_{\text{index}}$ measurement will show it. If index-induced instability is negligible at matched recall, the index-level claim fails and the contribution reduces to the embedding and attribution findings. We scale the bank to about a million items because CIRCLES found that retrieval quality improves with pool size. Exact search over a multi-view or set-valued representation costs a multiple of the bank size per query, which is where approximate graph search becomes necessary.
Why not just retrieve for each perturbed view and vote, as CRCA-Context does? This is a real threat, and it may match our best fix on stability. The difference we expect is cost: voting needs one traversal per view, while view-set search needs one Chamfer traversal. We will report both on a stability-versus-queries-per-second curve. If voting dominates at equal compute, we will recommend it. The attribution result stands either way, because it does not depend on which fix wins.
Why not just retrieve on text only, as DCoT recommends? Text-only retrieval ignores exactly the visual edits Plot Twist shows are damaging, such as grid lines and marker styles. Under those edits it stays stable only because it cannot see them, and CIRCLES argues this invites the similarity trap. Our perturbation split separates the two effects. We expect text-only keys to be stable under visual edits and unstable under nuisance-instruction and noisy-user rewrites, which CRCA-Context found most damaging.
Why not just use a bigger model? MATH-Vision's best systems sit near 30% against 68.82% for humans, and Plot Twist's title collapse appears across GPT-4V and Gemini Pro Vision. Scale does not remove this brittleness. Our attribution experiment is run at several model sizes, and it will show directly whether larger reasoners follow their demonstrations less and so become less sensitive to retrieval changes.
Proposed Experiments
Evaluation uses ChartQA-H test questions, Chart101's synthetic and real splits, and, as a harder transfer check, MathVista and MATH-Vision questions with a template-generated math demonstration bank. The metrics are clean accuracy, answer-change rate across perturbed pairs, collective accuracy, retrieval instability $I$, $\Delta_{\text{index}}$ at matched recall, robust Hit@3 with respect to the clean exact neighbors, and queries per second.
The first decisive check is the demonstration-swap experiment. If fewer than about a tenth of answer changes follow the swapped demonstrations, retrieval is not a meaningful cause, and the work ends as a clean negative result. The second decisive check is $\Delta_{\text{index}}$ at matched recall. A positive value that shrinks under GATE entry selection would be the first evidence that graph routing amplifies meaning-preserving query shifts. The third decisive check is the compute-matched comparison between view-set search and per-view voting. It settles whether the index-level fix is worth building or whether simple consensus is enough.
References
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- Retrieving Counterfactuals Improves Visual In-Context Learning
- CRCA-Context: Counterfactual Robustness of Repository Context Retrieval Under Equivalent Issue Descriptions
- Empowering Graph-based Approximate Nearest Neighbor Search with Adaptive Awareness Capabilities
- $\alpha$-Reachable Graphs for Multi-vector Nearest Neighbor Search
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- A Systematic Investigation of Commonsense Knowledge in Large Language Models
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset