Principle-Keyed Worked Examples for Expert Visual Reasoning

When a vision-language model fails an expert problem, such as a geometry contest question, a physics diagram, or a geography scenario, the failure is usually not that it misread the picture. On competition math, MATH-Vision attributes 42.2% of GPT-4V's errors to reasoning and only 31.9% to visual recognition. In GeoSQA, human annotators rewrote every diagram as text, and models still scored at chance. The missing piece looks like knowing which domain principle applies and how to bind it to this problem's quantities. Handing the model the principle's definition does not supply this: TheoremQA reports that appending theorem definitions to the prompt "did not provide a significant boost."

We propose to change what is retrieved and how it is retrieved. A small bidirectional encoder (a BERT-style model, which builds each word's vector from the words on both sides of it) reads the problem text together with a literal description of the diagram. It predicts which domain principle the problem calls for. The system then gives the reasoning model two or three worked applications of that principle, chosen to look unlike the test problem. The bet is that a worked application teaches the binding step a definition cannot, and that retrieval keyed on the hidden principle beats retrieval keyed on surface similarity. Existing exemplar retrieval for multimodal models uses surface similarity, and existing knowledge augmentation supplies definitions.

Core Claim

Expert visual reasoning splits into three stages: perceiving the diagram, identifying the governing principle, and applying it. Most of the recoverable error sits in the third stage. The third stage responds to worked applications retrieved by principle, but not to definitions, and not to exemplars retrieved by surface similarity. The claim is falsifiable at its root. If oracle principle labels plus worked applications do not beat definition augmentation and surface-similar exemplars, the idea is wrong and the identifier never needs to be built.

Research Program

The system has three parts, and the design isolates the contribution of each.

The first part is a perception pass. The vision-language model is asked only to transcribe the diagram: labels, quantities, relations, and axes, with no answer. This separates perception from reasoning in the same spirit as GeoSQA's human transcriptions, but automatically. It also lets us substitute GeoSQA's human descriptions as a perfect-perception control.

The second part is a principle identifier. It is a class-token attention probe in the style of Sina at SemEval-2023 Task 4. That system gives each label its own learned query vector, which attends over BERT's token vectors, and it detected abstract human values at 0.47 F1 against 0.28 for keyword extraction. Our change addresses data scarcity. TheoremQA covers 350 theorems in 800 questions, far too few examples per class to learn a query per theorem from scratch. We therefore initialize each principle's query from the encoder's own embedding of that principle's textbook statement. The score for principle $k$ is

$$ s_k = w_k^\top \sum_i \mathrm{softmax}_i\!\left(\frac{q_k^\top h_i}{\sqrt{d}}\right) h_i, \qquad q_k = W\,\bar e(\text{def}_k), $$

The symbols are as follows. $h_i$ is the encoder's vector for token $i$ of the problem plus diagram transcript. $d$ is the vector dimension. $\bar e(\text{def}_k)$ is the four-layer-averaged token vector of principle $k$'s definition, using the layer averaging of AMuSE-WSD followed by mean pooling. $W$ and $w_k$ are learned. Because new principles enter through their definitions, the probe can score principles it never saw in training. We also test domain-adaptive pre-training of the encoder on textbooks, following Pre-Training BERT on Domain Resources. That paper also warns of a specialization trade-off and of damage from incorrect rationales, so the worked-example library is the only place reasoning chains enter, and every chain there is verified.

The third part is a worked-application library. For each principle we curate two to four solved problems from textbook exercises, each with an explicit step that binds the principle's variables to the problem's quantities. The library is checked for overlap with every test item. At inference time we select the exemplars for the top-ranked principles that are least similar on the surface to the test problem. This prevents the model from copying a near-duplicate solution.

Baselines

The naive reference point is the reasoning model with direct answering and with chain-of-thought or program-of-thought prompting, given no retrieval at all. For reported numbers: GPT-4 with program-of-thought reaches 51.0% on TheoremQA, open-source models stay below 15% (TheoremQA), and GPT-4o reaches 30.39% on MATH-Vision against 16.97% for the open-source leader InternVL-Chat-V1-2-Plus (MATH-Vision). On GeoSQA, the best reported system is PMI retrieval over textbooks at 26.22%, and BERT scores 23–25% against a 25% chance rate (GeoSQA).

The definition-augmentation baseline replicates TheoremQA's theorem augmentation. It uses the same oracle or predicted principle, but supplies the definition text instead of worked applications. This is the ablation that isolates the "worked application versus definition" ingredient.

The strongest existing method for exemplar retrieval is the retrieval stage of DCoT. It retrieves the top three demonstrations by cosine similarity of text queries and improves ScienceQA by 3.0% over base models. We run it over our library, not its 80-pair general library, so that both systems see identical exemplars and differ only in how exemplars are selected. Beating it requires principle-keyed selection to win at the same exemplar count, the same context budget, and the same reasoning model.

The strongest competitor for the identifier itself is self-naming: the reasoning model is asked to name the governing principle, and that name is matched to the library. The Clinical Graph of Thoughts work shows that generative intermediate steps can carry large gains, raising 2-year outcome accuracy to 71.73% from ~52–56%, which makes self-naming a serious rival.

All methods share the same reasoning models (one closed and one open-source vision-language model), decoding settings, and prompt-tuning budget, tuned on a shared development split. The proposal counts as beating the field if principle-keyed worked applications from the learned identifier exceed both DCoT-style retrieval and definition augmentation by a margin outside bootstrap confidence intervals on at least two of the three benchmarks. The comparison against self-naming is reported as its own result, whichever way it falls.

Why not just…?

Why not just append the theorem definition, which is simpler? TheoremQA already ran this and found no significant gain. We rerun it under identical conditions so the worked-application effect is measured against it directly, not assumed.

Why not just retrieve similar solved problems with an off-the-shelf embedder? Surface similarity can surface near-duplicates whose solutions use a different principle. It can also miss same-principle problems that are worded differently, which is exactly the GeoSQA situation where adding scenario text lowered accuracy. Matched-library comparison against DCoT-style retrieval settles this. If surface retrieval ties principle-keyed retrieval, the identifier is unnecessary and we will say so.

Why not just let the reasoning model name the principle itself? This is the real threat, and it may work as well for frontier models. Two things argue for a separate identifier. Models show "blind faith in text," following misleading wording over evidence (Words or Vision), and Plot Twist shows accuracy collapsing from 79% to the 30% range when chart titles stop giving away the answer. A small trained probe may be less swayed by misleading surface cues, and it is cheap for open-source models that score below 15% on TheoremQA. We measure principle-identification accuracy for both methods, overall and on problems with misleading wording. If self-naming matches the probe, the contribution narrows to the retrieval and exemplar design, which still stands on its own.

Why not just fine-tune the vision-language model on more verified data? VeriSciQA shows that clean fine-tuning data helps, but its average gain across CharXiv, MMStar, and MathVista is +2.05%. That approach also requires retraining for each domain, whereas a library extends by adding entries. We include a fine-tuned open-source model trained on the same worked examples, to test whether retrieval at inference time does better than absorbing the same data into weights.

Proposed Experiments

The first decisive check is the oracle upper bound. On TheoremQA, which links questions to theorems, we give gold principle labels and compare four conditions: no retrieval, definition, surface-similar exemplars, and worked applications. We also add a wrong-principle control, which uses exemplars from a confusable principle. This separates "any worked example helps" from "the right principle helps." If oracle worked applications fail to beat definitions, the project stops here.

The second check moves to multimodal problems. We annotate a principle label for a MATH-Vision subset of about 500 problems, drawn from geometry and other principle-driven disciplines, with two annotators and reported agreement. On GeoSQA, we build principle labels from its textbook corpus. Running GeoSQA with both human diagram descriptions and our automatic perception pass measures how much perception errors limit the gain.

The third check measures the identifier itself. We report top-1 and top-3 principle accuracy under cross-validation held out by question, plus zero-shot accuracy on principles withheld from training. The probe is run with and without definition-initialized queries and with and without textbook pre-training. Pre-training is also tested across domains to see whether it reproduces the specialization trade-off. End-task accuracy is then reported as a function of identifier accuracy, which shows how much of the oracle gap a learned identifier recovers.

Error analysis uses the MATH-Vision error taxonomy, and should show the gain coming from fewer reasoning errors rather than fewer recognition errors. If instead recognition errors fall, the mechanism is something other than what we claim, and the paper must report that.