When a vision-language model solves a chart or geometry problem by writing code, the code usually runs. The answer is still wrong because the model misread the picture. It might take a bar as 40 instead of 45, or a tangent point as an intersection. Running the code catches crashes but not misreadings. Sampling more solutions does not help much either, because the same model tends to misread the same image in the same way every time. We propose to turn the model's reading of the figure into a program that can itself be executed. The model writes a scene description, such as a data table, a function, or a set of points and circles, and a plotting library re-renders it. The re-rendered image is then compared against the original. The system keeps several competing readings alive in a beam search (a search that keeps the top few partial candidates at each step instead of committing to one). It discards readings whose re-rendering disagrees with the input, and it solves the problem only on readings that survive.
The second ingredient addresses where alternative readings come from. We hypothesize that raising the sampling temperature produces readings that share the same error. Conditioning each branch on a different small set of worked demonstrations produces readings whose errors are less correlated. In-context learning means adapting a frozen model by putting examples in its prompt, and here it serves as a branch generator. Multimodal demonstrations are expensive in tokens, so we compress each demonstration set into an activation-level "task vector" following Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning. This lets each branch run as a cheap query-only pass. The claim to test is concrete: on figures that are renderings of structured objects, render-verified search over demonstration-diversified readings beats compute-matched best-of-N program sampling. It should win most clearly on the questions where the visual reading matters most.
Why the gap is in perception-to-structure, and why current fixes miss it
Diagnostic work motivates the problem. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset attributes 42.2% of GPT-4V's errors to reasoning and 31.9% to visual recognition. It also finds that natural-language chain-of-thought sometimes lowers accuracy for multimodal models. GeoSQA shows that replacing diagrams with human-written descriptions still leaves models failing, so the problem is not only pixels. Plot Twist shows models answering complex chart questions better than simple detail questions, which suggests they lean on textual priors over what the chart shows. Together these point to a failure at the conversion from what is seen to a structure that reasoning can operate on.
Program-based methods help but trust a single reading. Synthesize Step-by-Step linearizes charts into tables with DePlot and generates executable rationales offline, raising ChartQA human-split accuracy from 38.2% to 53.9% after fine-tuning. It uses one extraction and never checks it. PRISM separates perception from computation. It writes OpenCV grounding code and SciPy solver code and uses a critique agent to trigger repair loops. Its exploration is a single path, however, and its checker is an LLM. PRISM itself reports an "Oracle Gap": correct answers appear among samples far more often than majority vote selects them. That gap is exactly what a better selector should close.
Search methods supply the machinery but not this signal. V-ABS runs beam search over visual actions such as crop and rearrange, scored by an observer model. Its actions are fixed image operations, and its observer is learned, trained with an 80K-sample supervised set. EnCompass lets developers insert branch points into Python agent code and plug in beam search or tree search, which lifted GPT-4o on ARC-AGI from 24.0% to 38.3%. The scoring function is still left to the user. We reuse this style of infrastructure. Our contribution is the branch generator and the scoring function, not the search harness.
Core Claim
A structured figure has a property that general images lack: a correct reading can be checked by redrawing it. For a candidate scene program $s$ (for example, a table plus chart type and axis ranges, or a list of geometric primitives with coordinates), let $R(s)$ be its rendering and $x$ the input image. We score readings by
$$ \text{score}(s) = -\,d\big(R(s), x\big) + \lambda \cdot \mathbb{1}[\text{solver on } s \text{ executes and passes its assertions}], $$
where $d$ is a rendering discrepancy and $\lambda$ weights execution success. The discrepancy $d$ combines alignment-tolerant structural image distance, mismatch between text recognized in $x$ and text placed by $s$, and mark-level disagreements such as bar heights and line positions after axis calibration. This score is available before any reasoning is done. It therefore addresses the credit-assignment problem that the multi-path literature flags as open: intermediate steps are otherwise hard to score, and viable paths get pruned early. We make three claims. First, $d$ predicts whether an extraction is correct substantially better than asking the model to judge its own reading. Second, readings produced under different demonstration sets have less correlated errors than temperature samples. Third, the combination closes a meaningful part of PRISM's Oracle Gap at matched compute.
The claim is deliberately scoped to renderable figures: bar, line, and pie charts, function plots, and coordinate or synthetic geometry. We do not claim anything for natural images or hand-drawn physics scenes, and we report those separately as a boundary.
Research Program
The system has two programs with a fixed interface. The model first writes a scene program in a small typed schema, a JSON-like structure consumed by a renderer we write on top of matplotlib and a geometry drawer. It then writes a solver program that reads only the scene object. Fixing the interface is also a defense against the "partial migration" trap documented in A2DEPT: edits to how the scene is represented cannot silently break downstream functions, because the solver sees a schema-validated object.
Search proceeds in two stages. In the reading stage, $B$ branches each produce a scene program, each conditioned on a different demonstration set. The sets are retrieved or clustered to differ in figure style and reading conventions, for example stacked versus grouped bars or log axes. Readings are rendered and scored. The top-$k$ survivors are refined by feeding the model a visual diff, meaning the original and the re-rendering side by side with the highest-discrepancy regions marked. This makes the refinement loop consume execution feedback about perception rather than only tracebacks. In the solving stage, each surviving reading gets a solver program with execution retries. The final answer is chosen by score-weighted voting over readings, not over raw samples.
For the in-context component we use open models where activations are accessible, such as Qwen-VL-family and InternVL models. Each demonstration set is compressed into a multimodal task vector by the mean-activation-plus-head-selection procedure of MTV. In-Context Learning Creates Task Vectors explicitly leaves open whether multi-step, structured outputs can be carried by such vectors; its evidence covers single-token tasks. We therefore treat compression as a tested hypothesis rather than an assumption. Every experiment runs both with full in-context demonstrations and with task vectors, so that a failure of compression does not sink the search result.
Baselines
The naive reference is zero-shot direct answering by the same open model, alongside greedy single program-of-thought: one table-or-scene extraction, one solver program, one execution. This shows how much any structure helps before search.
The strongest simple competitor is compute-matched best-of-N program sampling with execution, where the answer is taken by majority vote among runs that execute. It is the obvious thing a practitioner would do, and it already benefits from execution feedback. We give it the same number of model calls and generated tokens as our full system, with rendering time reported separately since it runs on CPU. We tune its temperature and N on the same development split we use for our own hyperparameters. Beating it means a statistically significant accuracy gain (paired bootstrap over questions) on the full test split, with the gain concentrated on vision-essential items.
The strongest existing structured method is a PRISM-style single-path pipeline: perception agent, critique agent, solver, and repair loops, run with the same backbone and budget. On MATH-Vision, PRISM reports improving baseline accuracy from 42.8% to 59.5%, and on SeePhys-mini from 37.0% to 64.5% with Gemini 3.1 Flash Lite. These are closed-model numbers and not directly comparable to open 8B backbones. For ChartQA, the strongest figure in the file is the 53.9% human-split accuracy of Synthesize Step-by-Step (72.6% relaxed accuracy overall for its augmented MatCha), achieved with fine-tuning, which we do not use. For MATH-Vision the file reports GPT-4o at 30.39%. We found no reported result on Lacuna for training-free, execution-based pipelines on open 8B vision-language models on these benchmarks, so we will establish those numbers ourselves.
Two ablations isolate the key ingredients. Replacing $d$ with the model's own yes/no verdict on its reading isolates the render-and-compare signal. Replacing demonstration-diversified branches with temperature-diversified branches at equal branch count isolates the in-context contribution. We also report Pass@k over readings as the oracle ceiling, so that the fraction of the Oracle Gap closed can be stated directly.
Why not just…?
Why not just sample many programs and take a majority vote? This is the real threat, and it will win if misreadings are uncorrelated across samples. PRISM's evidence is that sampling fails on systematic visual errors, but PRISM compares against Majority@5 on a different model rather than at matched compute. We test the premise directly. For each question we measure how often all wrong temperature samples agree on the same wrong extracted value, and we compare that with demonstration-diversified branches. If temperature samples are already diverse and best-of-N matches our accuracy at equal budget, the search component is not worth its complexity. In that case we would report render-and-compare purely as a reranker over best-of-N samples, which is a smaller but still useful result.
Why not just let a critic model check the reading, as PRISM does? An LLM critic reads the same image with the same biases, and A2DEPT reports that LLM critics struggle with credit assignment across dependent steps. The verifier-swap ablation settles this empirically. We measure the area under the ROC curve for predicting extraction correctness against ground-truth tables on ChartQA and against synthetic geometry with known coordinates. If self-judgment matches the rendering discrepancy, the core claim fails.
Why not just put more demonstrations in the prompt instead of compressing them? For one branch this is fine and we run it. Across $B$ branches with image-bearing demonstrations, however, cost multiplies. MTV reports that 16-shot standard in-context learning, the most that fit, needed 20.6 GB and 3.3 minutes per 100 iterations, against 19.8 GB and 1.9 minutes for 400-shot-equivalent task vectors on VizWiz. We compare full-context and compressed branches at equal accuracy and report the cost ratio. If compressed vectors lose the structured-output behavior, which is plausible given the single-token scope of the original task-vector study, we keep full-context branches with fewer shots and report that negative result.
Why not just derender with a dedicated chart-to-table model and reason in text? This is the DePlot route used by Synthesize Step-by-Step. It is a strong single reading with no check. We include DePlot output as one extra branch in our beam. If it always wins the rendering score, the model-generated readings add nothing; if it sometimes loses, the verifier is catching its errors.
Proposed Experiments
The main testbeds are ChartQA, with emphasis on the human-written split where models reportedly score around 30% versus about 61% on machine-generated questions in ChartQA, and PlotQA. For geometry and function reasoning we use the renderable subsets of MathVista (FunctionQA and geometry sources) and of MATH-Vision (analytic geometry and graph-reading categories). SeePhys-mini's vision-essential split and the non-renderable remainder of MATH-Vision serve as the out-of-scope boundary. ChartQA's underlying data tables give ground truth for extraction correctness. For geometry we add a small synthetic set with known coordinates to calibrate $d$.
Three results are decisive. The first is verifier quality: the rendering discrepancy must separate correct from incorrect readings clearly better than self-judgment, or the project stops. The second is error decorrelation: demonstration-diversified branches must show lower rates of shared wrong readings than temperature branches, or the in-context component is dropped. The third is end-to-end accuracy at matched compute against best-of-N with execution and against the PRISM-style pipeline, broken down by vision-essential versus vision-optional questions. We expect the gains to sit almost entirely in the vision-essential stratum. A uniform gain would suggest we are only buying more compute, not better readings. As a secondary result, we report whether the visual-diff refinement loop keeps improving past the few-iteration plateau that Yan et al. observed for compiler-feedback loops. Our hypothesis is that it does, because its feedback targets the misreading rather than the symptom.
References
- Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- PRISM: Structured Decomposition for Multimodal Physics and Mathematical Reasoning
- V-ABS: Action-Observer Driven Beam Search for Dynamic Visual Reasoning
- EnCompass: Enhancing Agent Programming with Search Over Program Execution Paths
- $A_2$DEPT: Large Language Model–Driven Automated Algorithm Design via Evolutionary Program Trees
- In-Context Learning Creates Task Vectors
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation