Executable Programs as the Reasoning Interface for Small Vision-Language Models

Small vision-language models (systems that read an image and a question and write an answer) can usually see what is in a math diagram but still fail to reason about it. On competition math with figures, most of GPT-4V's errors come from the reasoning itself, not from misreading the image. This pattern is much worse in models under about 10 billion parameters. For them, writing out long step-by-step reasoning in natural language often hurts rather than helps, because every arithmetic or logical step is another chance to slip, and the slips compound.

This note proposes changing what the small model is asked to produce. Instead of a free-text reasoning chain, it writes a short program in two parts. The first part is a block of facts read off the image: lengths, angles, coordinates, table values. The second part is a solver that computes the answer using only those facts. The program runs in a sandbox, a locked-down Python interpreter with SymPy and a small geometry library. The model is then trained with reinforcement learning, where the reward comes from running the program rather than from a judge model. The scientific question is whether this interface lowers the model size at which multi-step visual math reasoning "switches on." The practical question is whether a 2–8B model trained this way can match modular systems that pair a 7B perceiver with a separate 8B or larger text reasoner.

Core Claim

A simple model explains the claim. Suppose a problem needs $n$ reasoning steps. Accuracy then behaves roughly like

$$\text{acc}(n) \approx p_{\text{read}} \cdot q^{\,n},$$

where $p_{\text{read}}$ is the probability of extracting the needed visual facts correctly and $q$ is the per-step success rate. This is the compound-probability account of emergence summarized in the research file: small models have low $q$, so $q^n$ collapses and accuracy looks flat until scale raises $q$.

In natural-language reasoning, $q$ covers two things: choosing the right operation and carrying it out without an arithmetic or algebra slip. An executable program removes the second failure, because the interpreter does the carrying out. What remains is the model's ability to state each step correctly as code. We therefore predict three things. At a fixed model size, the fitted $q$ will be higher for the program interface than for text reasoning. The advantage will grow with $n$. And the smallest model size that clears a fixed accuracy bar on multi-step problems will be lower, which means the reasoning threshold moves.

A second claim concerns the image. The fact block is required, and it is checked against the image wherever ground truth exists. This should make the model's answer depend on the image rather than on shortcuts in the question text. That matters because MathVerse found some models score over 5% higher when the diagram is deleted.

What Exists and Where This Differs

Three lines of prior work motivate the proposal by documenting the gap. MATH-Vision attributes 42.2% of GPT-4V errors to reasoning and 31.9% to recognition. It also reports that chain-of-thought prompting lowers accuracy for several multimodal models. Measuring Taiwanese Mandarin Language Understanding reports the same kind of reversal for small text models. Decoupling Reasoning and Perception shows that long reasoning chains drift away from the image into invented details.

The closest competitors decouple seeing from reasoning, but they pass information through natural language. RAPID uses reinforcement learning to train a Qwen2.5-VL-7B perceiver to write captions that help a frozen text reasoner answer correctly. The dialogue system of Jia et al. lets a text model question a vision model and needs no training. Both leave the actual computation to a second, often larger, language model that reasons in prose. Our proposal hands the computation to an interpreter and keeps a single small model.

The visual-programming line supplies the interface idea but does not train with execution feedback. ViperGPT has Codex write Python that calls vision tools, with no training at all. VDebugger trains a separate critic that repairs visual programs from line-by-line execution traces. It also reports that roughly 58% of failures are program-logic errors, and that zero-shot self-debugging fails in small models. ChartPaLI-5B and Synthesize Step-by-Step use Python only in two ways: as a refinement step after inference, or to verify synthetic training data offline.

Execution-rewarded reinforcement learning already works in text and in narrow visual domains, and we reuse its method. PPOCoder and AutoTriton train compact code models against compilers and unit tests. CAD-RL does the same for CadQuery scripts, rewarded by 3D overlap. However, CAD-RL keeps GPT-4o in the reward loop. Decoupled Physical Modeling splits physics solutions into a modeling part and a computation part, but scores the modeling part with a Gemini judge.

What no prior work does is study how the benefit of a program interface changes with model size. Specializing Smaller Language Models supplies the precedent for this kind of question. It showed that task-focused distillation turns the flat below-threshold curve into a log-linear one, but only for text reasoning and without execution. Code Execution as Grounded Supervision shows that execution-derived traces cut reasoning length by 20–30% and raise the correctness of intermediate steps. That work also has no images and no online reinforcement learning.

Research Program

We will build a sandbox that runs a restricted Python dialect with no network access and hard time and memory limits. It provides SymPy, exact rational arithmetic, and a small construction library for points, lines, circles, intersections and a numeric solver. The model's output must contain a typed fact block, whose entries each carry a unit or type. It must also contain a solver function that reads only from the fact block. A static check rejects solvers that contain unexplained numeric constants, which closes the easiest route for smuggling in a guessed answer.

Training data comes from two sources. The first is procedurally generated diagrams covering geometry constructions, function plots, and bar and line charts. Because each image is rendered from a known scene description, we have ground truth for both the facts and the answer. We also control the number of reasoning steps $n$ exactly. The second source is the real-image question-answer pairs in ViRL39K from RAPID, which have answer labels only.

The reward has three terms, all deterministic. The main term is whether the executed answer is correct. On synthetic data, a fact-accuracy term scores each extracted value with a tolerance. A small format term rewards programs that run. No language-model judge is used anywhere.

Training runs in two stages. A cold-start fine-tuning stage uses execution-verified programs on synthetic problems. It is followed by Group Relative Policy Optimization, the group-baseline policy-gradient method used by AutoTriton. We run this recipe at five sizes, from roughly 2B to 8B, drawn from the Qwen2.5-VL and Qwen3-VL families.

One known constraint shapes the lower end of the study. Koutcheme et al. find that code abilities decouple below about 2B parameters. We therefore treat the smallest size as a possible floor for the method itself, not as a guaranteed success.

Baselines

The naive reference point is each base model answering directly, with and without chain-of-thought prompting. This establishes whether prompting helps or hurts at each size. MATH-Vision's random-chance level of about 5.86% and GPT-4o's 30.39% anchor the scale.

The most important ablation is text-CoT GRPO. It uses the same base models, the same data, the same number of rollouts and the same tuning budget. The only difference is that the model writes free-text reasoning and is rewarded on final-answer match. This isolates the interface, which is the proposal's key ingredient. The research file contains no reported result for text-CoT GRPO on ViRL39K at 2–8B sizes, so we run it ourselves. Two more ablations remove one component each. One keeps the program interface but drops the fact-accuracy reward. The other keeps the programs but trains by fine-tuning only, in the style of Code Execution as Grounded Supervision.

The strongest existing competitor is RAPID. Its paper reports a 53.2% multi-benchmark average accuracy with a Qwen2.5-VL-7B perceiver and a Qwen3-8B reasoner. That is an 11.2-point gain over the base model, which is therefore about 42.0%. RAPID reaches 58% with a 120B reasoner. For a fair comparison, we rerun RAPID on our evaluation suite and compare at matched total parameter count: our single 8B model against RAPID's roughly 15B pair. We also give RAPID's reasoner a Python tool, so the comparison tests our trained, fact-committed interface rather than the mere presence of an interpreter.

On the separate MathVision and MathVerse benchmarks, the reference is the training-free 72B+32B system of Jia et al., which reports 52.6 and 54.4 respectively. On ChartQA, the reference is ChartPaLI-5B with program refinement, reported at 81.3% by Chart-based Reasoning. Three results would count as beating the baselines. First, an 8B model that matches or exceeds RAPID's 53.2% on RAPID's own benchmark average. Second, a consistent margin over text-CoT GRPO on the multi-step subset at every size of 4B and above. Third, an emergence size at least one model-size step smaller than text-CoT GRPO's.

Why not just…?

Why not just run standard GRPO with free-text reasoning, the now-standard recipe? This is the real threat. It may be that reinforcement learning, not the interface, does all the work. The text-CoT GRPO ablation settles the question directly. If the fitted per-step success rate $q$ and the emergence size do not differ between the two interfaces, the central claim fails, and we will report it that way. The evidence that the interface might matter comes from two findings. Execution-grounded traces raise intermediate-step correctness from 73.0% to 91.5% on LiveBench (Code Execution as Grounded Supervision). And free-text reasoning in multimodal models loses contact with the image as it grows longer (Jia et al.).

Why not just caption the image and give the caption to a strong text model, as RAPID does? RAPID is strong and cheap, and it may win on accuracy at equal total parameters. However, its text interface cannot be checked. GeoSQA reports that even human-written text descriptions of diagrams leave models failing at the reasoning. Our comparison against RAPID with a tool-equipped reasoner separates two factors: having an interpreter at all, and training the perceiver to commit to typed facts. If RAPID with a tool matches us, the contribution shrinks to data and reward design.

Why not just prompt a small model to write a program at inference time, or apply program-of-thought refinement as ChartPaLI does? VDebugger reports that small generators produce mostly logic errors, and that zero-shot self-repair does not fix them. Weng et al. find that backward verification is absent below scale thresholds. We include zero-shot program prompting at each size, and the program-SFT-only ablation, to measure how much of the gain requires the reinforcement-learning stage.

Proposed Experiments and Decisive Checks

Main benchmarks. We evaluate on MATH-Vision stratified by its five difficulty levels, on MathVerse across all six text-redundancy variants, on MathVista, on ChartQA, and on PhysReason. For PhysReason, the file's reported numbers for Qwen2.5-VL-3B are 13.3% for the base model and 22.9% after decoupled physical modeling with GRPO (Decoupled Physical Modeling).

Test 1: step count. On held-out synthetic problems with $n$ from 1 to 8, we fit $p_{\text{read}}$ and $q$ separately for each interface and size. The prediction is a higher $q$ for programs, with the gap widening as $n$ grows. If the curves for the two interfaces run parallel in $n$, the threshold story is wrong.

Test 2: emergence size. We define it as the smallest size that exceeds the midpoint between chance and the 8B ceiling on MATH-Vision levels 4–5. The prediction is that the program interface reaches this bar at a smaller size than text reasoning.

Test 3: counterfactual diagrams. We re-render synthetic diagrams with one fact changed and check, by execution, whether the answer changes as the ground-truth solver says it should. We also compare MathVerse vision-only accuracy against text-dominant accuracy. The prediction is that models trained with the fact reward follow image edits far more often, and have a smaller vision-only drop, than text-CoT GRPO does.

Test 4: cost of specialization. We report MMStar and MM-Vet before and after training, because specialization is known to cost general ability. A larger drop than text-CoT GRPO would be a real cost.

What would falsify the proposal. Any one of three results would do it. The first is no difference in $q$ between interfaces at matched compute. The second is no shift in emergence size. The third is RAPID with a Python tool beating our 8B model at equal total parameters.