When a vision-language model gets a diagram-based math problem wrong, the cause can be very different from one problem to the next. Sometimes it misread the picture. Sometimes it did not know the formula. Sometimes it knew everything and simply composed the steps badly. Today's adaptive retrieval systems ask only one question, "does the model know enough, or should it look something up?", and they answer it by fetching text documents. In visual reasoning this question is usually the wrong one. On elementary visual math, about 62% of Gemini 2.0 Flash's errors are perception errors (VCBench). On competition problems, the largest share of GPT-4V's errors (42.2%) are reasoning errors made after the image was read correctly (MATH-Vision). Fetching a textbook passage repairs neither.
We propose a trigger that predicts, at each reasoning step and before an error happens, which kind of help the step needs. It chooses among four options: looking again at a region of the image, retrieving a concept or formula, handing the arithmetic to a code executor, or doing nothing. The failure categories come from hallucination taxonomies. Taxonomy work has so far used them only to classify errors after the fact; here they become routing labels at inference time. We label each failure type operationally, by which oracle fix actually repairs the step. This sidesteps the familiar problem that early misreadings and later logic failures get tangled together when humans try to assign blame after the fact.
Core Claim
Our claim is that, for multimodal reasoning, the value of retrieval triggering lies in choosing what kind of intervention to make, not in deciding whether to retrieve. We test three concrete consequences. First, an untyped trigger (retrieve-or-not) will waste calls and will sometimes hurt on steps whose real problem is perceptual. Second, a cheap typed gate can recover most of the gain that an oracle router achieves. Third, it can do this with far fewer interventions than an "apply every tool at every step" policy. If oracle typed routing does not beat oracle "apply everything", the idea is wrong, and we say so up front.
Relation to Prior Work
The text-only adaptive-retrieval literature supplies the gating machinery. Self-RAG trains a model to emit a retrieve/no-retrieve token. Thrust scores queries by how their hidden states sit relative to cluster centroids, at about 0.001 seconds per query. GECE flags long-tail queries using calibration and gradient signals. Chain-of-Action asks the model to emit explicit "missing" flags per sub-question, which cut StrategyQA tokens from 82,120 to 30,485. All four make a binary decision and are evaluated only on text. We reuse Thrust-style hidden-state probing and Chain-of-Action-style self-flagging as components and as baselines. Our change is the output of the decision: a fix type rather than a yes/no.
The closest multimodal work treats retrieval as uniform. AR-MCTS runs a tree search over reasoning steps and retrieves from a mixed text-and-image corpus at every node expansion. It gains 8.3% on We-Math with Qwen2-VL-7B, but it never asks whether a step needed retrieval at all, or needed a different fix. DCoT classifies each question into one of eight categories, yet still retrieves three demonstrations for every query, which is a fixed rather than adaptive policy. CofactGVR grounds answers in image regions and checks causally whether each region matters, but grounds every instance and never retrieves knowledge. Our "look again" fix borrows its region-grounding idea, and our compute fix borrows the executable rationales of Synthesize Step-by-Step.
Three lines of evidence motivate the typed design. Error taxonomies show that perception, knowledge, and computation failures are separable and have different frequencies: VCBench, MATH-Vision, and the knowledge/extraction/computation split of MedCalc-Bench. WE-MATH supplies the hierarchical scaffold, with step-level sub-problems tagged by knowledge concept and static "knowledge cards". It reports that GPT-4o falls from 72.8% on one-step problems to 43.6% on three-step problems. Finally, Choudhary et al. and Wang et al. show that unnecessary retrieved context can actively induce errors.
Two results constrain the design. GeoSQA shows that perfect human transcription of a diagram still leaves models failing, so "convert the image to text" cannot be the only fix. Plot Twist shows models answering complex chart questions better than elementary readings of the same chart, partly by leaning on the chart title. A model's success on a hard question is therefore not evidence that it perceived the chart correctly, and our labels must not assume it is.
Research Program
Labels: classifying each failure by which oracle fix repairs it. We take step-level sub-problems from WE-MATH, which already decomposes problems into a hierarchy running from perception, to concept application, to composition. For each step the base model fails, we rerun the step under three oracle fixes, one at a time. The first injects corrected perceptual facts: annotators fix the values the model read from the diagram, on a subset of roughly a thousand steps to keep annotation cost bounded. The second injects the WE-MATH knowledge card for the step's concept. The third replaces the model's arithmetic with an exact executor. A step's label is the set of fixes that rescue it; when several do, we take the cheapest single fix. Steps rescued by none are labeled "composition". Steps the model already gets right, and that any fix would break, are labeled "harmful to intervene". This last label is what lets us measure distraction directly.
The gate. At each step the gate picks a fix $r$ from {none, look again, retrieve concept, execute}. Here $\hat p(\text{error}\mid r, s)$ is the predicted chance that step $s$ is still wrong after applying $r$, $c(r)$ is the fix's cost in tokens and calls, and $\lambda$ sets the accuracy/cost tradeoff. The gate chooses
$$ r^*(s) = \arg\min_{r}\; \hat p(\text{error}\mid r, s) + \lambda\, c(r). $$
We build two versions. The white-box gate is a small probe on the model's hidden states at the step boundary, in the spirit of Thrust, trained on the oracle-fix labels. The black-box gate, for API models, uses three stability tests, each aimed at one failure type. Answer stability under non-semantic image changes (re-rendering, gridlines, cropping, following Plot Twist's perturbations) is the perception signal. Stability under paraphrasing the concept is the knowledge signal. Agreement across sampled reasoning chains is the composition and computation signal. The black-box gate is a small classifier over these three features.
The fixes. "Look again" crops and zooms into the region the model cited, then re-queries the model on that region. "Retrieve concept" queries a concept corpus assembled like AR-MCTS's mixed corpus, plus the WE-MATH concept cards. "Execute" asks the model to write the step as Python and runs it. The fixes are deliberately simple, so that any gains come from routing and not from a better tool.
Baselines
The naive reference points are the base model with plain chain-of-thought and no fixes at all, and the opposite extreme: apply all three fixes at every step. The "apply all" baseline sets the accuracy ceiling for untyped help, and it is the comparison that determines whether routing matters. To beat it, typed routing must match its accuracy within noise while using at least 50% fewer fix calls, or beat its accuracy outright by avoiding harmful interventions.
The strongest existing method is AR-MCTS. It retrieves at every tree expansion and trains a step-scoring reward model. The file reports a +8.3% gain on We-Math for Qwen2-VL-7B and +12.5% (math) and +7.7% (physics) for GPT-4o on GAOKAO-MM (AR-MCTS). We run it with Qwen2-VL-7B on the same corpus and the same search budget (number of expansions). We also run a variant where our gate decides at each node what to do, in place of always-retrieve. Beating AR-MCTS means either exceeding its We-Math gain at equal search budget, or matching it with substantially fewer retrieval calls.
We also run the untyped gates under identical tool access, and these are the ablation that isolates our key ingredient. We include a Thrust-style probe trained on the same labels collapsed to "any fix helps vs. none"; when it fires, it invokes either the concept retriever or all three fixes. We also include Chain-of-Action-style missing flags that are explicitly untyped. For Self-RAG, Thrust, and GECE we found no reported result on Lacuna for any multimodal benchmark, so we reimplement them rather than quote numbers. WE-MATH's static knowledge-card augmentation serves as an oracle-knowledge reference; no reported KCA accuracy is in the file, so we measure it ourselves. Every gate gets the same training labels, the same tuning budget (a fixed number of validation sweeps over the cost weight $\lambda$), and the same tools.
Why not just…?
Why not just apply every fix at every step? If tools were free and never harmful, this would be the right answer, and it is the main threat to the proposal. The evidence that it falls short is indirect but specific. Irrelevant retrieved context induces errors in text (Choudhary et al.), and AR-MCTS pays search overhead for retrieval on every node. Our "harmful to intervene" label measures directly how often a correct step is broken by an unneeded fix. If that rate is near zero and cost is ignored, "apply all" wins on accuracy. In that case our contribution narrows to efficiency, and we will report it that way.
Why not just transcribe the image to text and then run ordinary text-based adaptive retrieval? Two results in the file argue against this. GeoSQA shows that even perfect human transcription leaves reasoning failures intact, so perception is not the only bottleneck. Plot Twist shows that model-produced readings of elementary chart values are unreliable: collective accuracy across all elementary questions on a chart is below 25% for every model tested. Our "transcribe then text-only Self-RAG/Thrust" baseline tests this pipeline directly. We expect it to do well on knowledge-labeled steps and poorly on perception-labeled ones.
Why not just prompt the model to say which kind of help it needs? This is a serious competitor. It extends Chain-of-Action's missing flags to typed flags and requires no training. We include it as a baseline. If prompted self-diagnosis matches the probe and stability gates on routing accuracy, the contribution reduces to "typed routing helps, and models can do it themselves". That would still be a useful finding, and we would report it as such. Our reason to doubt it: the "Harder is Easier" paradox suggests models are poorly aware of their own perception failures.
Proposed Experiments
We train gates on WE-MATH steps and test them on held-out WE-MATH problems, reporting accuracy and the four WE-MATH behavior categories: insufficient knowledge, inadequate generalization, complete mastery, and rote memorization. We expect typed routing to shift problems from insufficient knowledge to complete mastery. We then transfer the gates without retraining to MathVista (GPT-4V reference 49.9%), MATH-Vision (GPT-4o 30.39%), and VCBench (Gemini 2.0 Flash 49.77%, Qwen2.5-VL-72B 48.08%, GPT-4o 40.29%). For each benchmark we report accuracy, fix calls per problem, tokens, and wall-clock time. Models are Qwen2-VL-7B and Qwen2.5-VL-72B for both gates and GPT-4o for the black-box gate only.
Three checks decide the outcome. The first is the oracle comparison: oracle typed routing versus oracle "apply all", on the annotated subset. If typed routing gives no accuracy or harm advantage there, we stop. The second is a breakdown of routing accuracy by failure type. We expect the gate to send perception-labeled steps to "look again" on VCBench, where perception dominates, and composition-labeled steps to "none" or "execute" on MATH-Vision. If the gate instead sends most steps to concept retrieval on both benchmarks, it has learned only the generic retrieve-or-not signal. The third is a harm audit: among steps that were correct without help, how often each policy breaks them. This directly tests whether untyped retrieval actually distracts models in the visual setting.
References
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Thrust: Adaptively Propels Large Language Models with External Knowledge
- On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models
- Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models
- Progressive Multimodal Reasoning via Active Retrieval
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- CofactGVR: Counterfactual Intervention for Grounded Visual Reasoning
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
- We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
- Grounding in social media: An approach to building a chit-chat dialogue model
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details