Principle-Keyed Exemplars for Multimodal Science Problems

Vision-language models, meaning models that read an image and text together and answer in text, solve competition math and physics with diagrams poorly. GPT-4o reaches 30.39% on MATH-Vision against 68.82% for humans, and 13.10% on OlympiadBench physics. The usual fixes each seem to fail on their own. Handing the model the relevant theorem's textbook definition did not help on TheoremQA. Adding a few worked examples to the prompt makes accuracy swing with whichever examples happen to be chosen. Retrieving examples by image similarity tends to teach the model to copy answers rather than reason.

This note proposes one concrete fix. A model gets stuck because it does not know how to bind the quantities in this diagram to the premises of the right principle. The worked examples placed in front of a test problem should therefore be chosen by the physical or mathematical principle the solution uses, not by how similar the picture or question looks. Each example should show that binding step explicitly, as a short executable program. The claim has two parts. Principle-keyed examples should raise accuracy where definition injection does not. They should also shrink the spread of accuracy across different example draws, and especially lift the worst case. Both parts are falsifiable with existing benchmarks and one control condition.

Core Claim

Four strands of evidence suggest a single missing ingredient.

The first strand is error analysis. On MATH-Vision, the largest share of GPT-4V's errors (42.2%) are reasoning errors made after correct perception (Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset). Models also fail even when diagrams are replaced by human-written text (GeoSQA). The bottleneck is organizing what was seen into a deduction.

The second strand concerns definition injection. TheoremQA found that putting theorem definitions in the prompt gives no significant boost. Its best result, GPT-4 at 51.0%, came from Program-of-Thought prompting, where the model writes Python that an interpreter executes. That paper left open why definitions fail. A natural reading is that a definition states the rule but never shows how to map a concrete problem onto its premises.

The third strand concerns how examples are retrieved. What Makes Multimodal In-Context Learning Work? shows that retrieving visually similar examples with CLIP embeddings mostly induces label copying and recency bias. DCoT found that text-only retrieval beats image-embedding retrieval because image features add noise.

The fourth strand is example sensitivity. Few-shot results are highly sensitive to which examples are chosen (On sensitivity of meta-learning to support data; Monte Carlo Sampling for Analyzing In-Context Examples). One bad example can push accuracy below zero-shot.

Our hypothesis ties these together. A worked example helps when it carries the transferable step, which is the principle together with its binding to givens. It hurts when it carries surface features the model can copy. Keying retrieval on the principle and writing the binding as code should therefore help on both fronts at once. It should give higher mean accuracy than either definitions or surface-similar examples. It should also give lower variance, because every admissible example carries the same useful content.

The closest existing system is ContextNav. It uses an agent to retrieve, denoise, and rewrite multimodal examples to match the target question's format, and reports an average 16.8% in-context-learning gain across six models. ContextNav selects examples by semantic similarity and aligns their surface format. It uses no domain principle index, costs about 22.51K tokens and 3.26 seconds per query, and breaks down with small agent models. We propose the opposite design point: a fixed, non-agentic pipeline whose selection key is domain knowledge, plus a control that separates "right principle" from "right format." CAMS balances example difficulty and model uncertainty, but it scores complexity from text captions and draws from a closed general-VQA pool. We reuse its finding that difficulty balance matters, but treat difficulty as a secondary key within a principle bucket.

Research Program

The example bank. We build the bank from the expert step-by-step solutions in OlympiadBench. We split it by source competition and year into a bank half and a test half. We remove any test problem whose question text or diagram is near-duplicated in the bank.

Annotating each bank problem. Each bank problem receives two annotations. The first is one or more principle tags, such as conservation of angular momentum, the power of a point, or Kirchhoff's voltage law. An LLM proposes these from the expert solution, and they are normalized into a closed vocabulary by merging synonyms. A random sample is audited by hand for tag accuracy. The second annotation is a structured diagram parse: a text list of entities, given quantities, and relations, such as "block A, mass m, on incline angle θ, string over pulley to B." Following DCoT's finding, we never put raw image embeddings into the retrieval key. Each expert solution is then rewritten into a fixed three-part program: extract givens from the parse, assert the premises of the tagged principle with those givens bound, then compute. The program is kept only if it executes to the expert answer.

Answering a test problem. The model first produces the diagram parse and a ranked list of candidate principles. The system then retrieves K examples (K of 3 by default, following DCoT's optimum) from the top principle bucket, spreading their difficulty in the manner of CAMS. The model writes and executes a program in the same three-part format. A single fallback is allowed: if execution fails, the second-ranked principle is tried once. Excluding that rare fallback, this is two model calls per problem, with no agent loop.

Measurement. Beyond mean accuracy, we report what Agarwal et al. reported for support sets. For each problem we draw 20 different example sets from the admissible pool and report worst-draw, median, and best-draw accuracy per condition. We then re-run the MATH-Vision error taxonomy on a hand-labeled sample to see whether gains come from fewer reasoning errors, as predicted, or from fewer vision errors, which would contradict the mechanism.

Baselines

The naive reference points are zero-shot chain-of-thought and zero-shot Program-of-Thought. For GPT-4o on OlympiadBench, the reported full-benchmark figures are 25.89 average, 32.48 math, and 13.10 physics (OlympiadBench). On MATH-Vision the reported figure is 30.39% (MATH-Vision). Our test half is a subset, so all conditions are re-run on it rather than compared against the published numbers.

The second baseline is definition injection: the gold principle's textbook statement is added to a zero-shot Program-of-Thought prompt. It reproduces TheoremQA's "theorem augmentation" in the multimodal setting. No numeric delta for that method is reported in the file; it is described only as not significant.

The third baseline is surface-similar retrieval with the same K, the same program format, and the same bank. We run it in two variants: text-only question similarity, as in DCoT, and CLIP image-and-text similarity, as in the setup Baldassini et al. studied.

The strongest existing method is ContextNav, run on the same bank and the same test problems with its own agent. Its 16.8% figure is an average gain across eight general benchmarks. The file reports no per-benchmark MathVision number and no OlympiadBench number for ContextNav, so we must measure it ourselves. We will report its token and latency cost next to ours.

The key ablation is a format-matched wrong-principle control. It keeps everything identical, including the three-part program format and difficulty spread, but draws the examples from a deliberately different principle bucket. An oracle variant that uses the gold principle tags bounds how much is lost to errors in principle prediction.

Fairness. All conditions use the same model, decoding settings, and total number of model calls. Any method that samples more gets the same budget handed to the baselines as self-consistency, meaning majority vote over repeated samples. Prompt templates receive equal tuning effort on a held-out development slice of the bank half.

What counts as success. Beating the baselines means two things. Principle-keyed examples must exceed both surface-similar retrieval and definition injection on mean accuracy on the physics multimodal subset. They must also raise worst-draw accuracy by more than they raise best-draw accuracy, at no more than one fifth of ContextNav's tokens. Matching ContextNav at that cost would count as a positive result. Losing to it on accuracy would bound the claim to an efficiency result.

Why not just…?

Why not just scale or use a stronger model? Frontier models remain far below human performance on exactly these benchmarks: 30.39% against 68.82% on MATH-Vision, and 13.10% on OlympiadBench physics. We run the method on both an open model (Qwen2.5-VL) and GPT-4o. If the gain disappears on GPT-4o, we will report that the method is a small-model aid only.

Why not just use self-consistency over randomly drawn examples? This is the most serious threat to the variance claim. Averaging over many random example sets also narrows spread, and it needs no index. We give it the same number of model calls our method uses and the same program format. If it matches our worst-draw accuracy, then principle keying contributes nothing beyond averaging and the variance claim is withdrawn. The mean-accuracy claim would then rest only on the wrong-principle control.

Why not just use ContextNav's agentic alignment? ContextNav's own ablation shows that format alignment carries much of its gain: removing it drops the gain from 11.8% to 6.7%. That raises the possibility that format is everything. The wrong-principle control tests exactly this. If wrong-principle examples in the right format perform as well as right-principle ones, our idea reduces to a cheap version of ContextNav's alignment. We would say so plainly.

Why not just give the theorem definition plus Program-of-Thought? TheoremQA reports that definitions did not help significantly. We still run this condition with the gold principle, so the oracle comparison isolates the effect of a worked binding over a bare statement.

Why not just fine-tune on the bank's solutions? For open models, this is a strong competitor on mean accuracy. We include a low-rank-adapter fine-tune of the open model on the bank programs. If it wins clearly, the training-free claim becomes mainly about closed models and per-query variance.

Proposed Experiments

Main experiment. The main experiment covers the OlympiadBench test half (math and physics, multimodal problems separated from text-only) and the full MATH-Vision set. MATH-Vision gives a second domain with an existing error taxonomy, and its examples are drawn from a bank built the same way from MATH-Vision competition sources held out by competition. TheoremQA serves as a text-and-diagram check of whether worked bindings succeed where its definition augmentation failed.

Decisive check 1: content versus format. This compares right-principle with format-matched wrong-principle examples. The idea survives only if right-principle examples win by a margin larger than the spread across example draws.

Decisive check 2: variance. This compares worst-draw accuracy under principle keying with surface keying and with call-matched self-consistency.

Decisive check 3: mechanism. This tests whether the gain shows up as fewer reasoning errors rather than fewer vision errors in the taxonomy. If gains instead track the oracle-versus-predicted principle gap, the next bottleneck is principle recognition from diagrams. That would redirect the work toward better diagram parsing rather than better retrieval.