Render-and-Reread: Test-Time Training for Chart Reasoning from Self-Checked Data

Vision-language models, which answer questions about images, often fail at reading charts in ways that look like reasoning failures but start as perception failures. They copy answers from chart titles instead of reading the bars. They lose accuracy when a grid line is added. They struggle to answer every elementary question about the same chart correctly. The usual fix is to generate millions of synthetic training examples offline with a large model, then train a smaller model on them. That fix is expensive, and the resulting model is still frozen when it meets a chart style it has never seen. We propose a different fix. At test time, the model writes down the data it believes a chart contains. A plotting program redraws that data. The model accepts its own reading only if the redrawn chart matches the original once the title and other text are hidden. It then trains briefly on a few dozen questions computed from the accepted data, asked about the real test chart, and only after that answers the user's question.

The idea joins three threads. The first is test-time training: updating a model on each unlabeled test input using a self-made learning task. The second is the multimodal reasoning gap, where perception errors grow into confident multi-step mistakes. The third is the cost of synthetic data generation. Here the synthetic data is made just in time, per query, and checked by a renderer rather than by a teacher model. If the idea works, a small open model can adapt to a new plotting style or a misleading title without an external judge and without an offline corpus of millions of examples.

Core Claim

The self-supervision signal for adapting a chart-reading model at test time should come from executable consistency between the model's extracted data and the pixels. It should not come from agreement among the model's own answers. Answer agreement inherits the model's shortcuts. When the title states a value, many sampled answers agree on that value whether or not the bars support it. A redrawn chart with the title removed cannot be fooled this way.

This differs from the closest existing work in a specific way. Efficient Test-Time Scaling for Small Vision-Language Models adapts a small model at test time using pseudolabels, meaning answers the model assigns to itself. Those pseudolabels come from high-agreement answers over brightness, rotation and paraphrase perturbations, and the weights are reset after each query. Our pseudolabels come from an external, non-learned check, and they are elementary data-reading facts rather than the final answer. DiSCTT also routes test-time updates by trajectory agreement, but works only on text. Reactivating Test-Time Scaling for Plane Geometry Problem Solving uses agreement across symbolic, program and chain-of-thought traces to stop sampling early, but never updates weights and relies on a large offline synthesis stage. We take the general idea of turning one test input into a self-supervised problem from Test-Time Training with Self-Supervision. We take the reconstruction-as-supervision idea from Test-Time Visual In-Context Tuning, which showed that one self-supervised update on one corrupted image can beat training on 64 labeled target examples, but only for dense low-level vision tasks.

Research Program

The procedure for one chart $x$ with question $q$ has three stages.

Stage 1: sample candidate readings. The model samples $N$ candidate readings. Each reading is a linearized data table plus a short style description: chart type, axis ranges, series colors and whether grid lines are present.

Stage 2: render and score. A deterministic plotting script renders each candidate $\hat T_i$ as an image $r(\hat T_i)$. A non-learned agreement function $s(x, r(\hat T_i))$ scores the match. It compares edge and color-mask maps of the two images after an off-the-shelf text detector has masked all text regions, including titles, legends and tick labels. Masking text is deliberate: it makes the check blind to the title shortcut documented in Plot Twist. If the best score falls below a threshold, the model does not adapt and answers zero-shot. This abstention matters because a wrong pseudolabel is worse than none.

Stage 3: train briefly, then answer. From the accepted table $T^\ast$, a template program generates $K$ elementary questions with exact answers: maximum, minimum, value at a category, difference between two series, and axis range. For a small set of low-rank adapter weights $\phi$ (a small trainable add-on to the frozen model), we minimize

$$ \mathcal{L}(\phi) = \sum_{k=1}^{K} \ell\big(f_\phi(x, q_k), a_k\big) + \lambda \sum_{j=1}^{J} \ell\big(f_\phi(r_j(T^\ast), \text{table}), T^\ast\big), $$

Here $(q_k, a_k)$ are the generated question-answer pairs asked about the real test chart, $\ell$ is token cross-entropy, $r_j$ are $J$ re-renderings of $T^\ast$ in randomized styles, and $\lambda$ weights this table-reading anchor term. After a few gradient steps, the model answers $q$.

We run the procedure in two modes. In single-query mode the adapter is reset after each query, as in TTAdapt. In online mode, adapter weights carry over across a stream of charts from one new source, so the cost is amortized when the shift persists. This mirrors the online variant of Test-Time Training with Self-Supervision.

Generating these examples costs one renderer call per candidate reading and no large-model labeling. We will report this cost directly against the offline corpus of Synthesize Step-by-Step.

The research question we most need to answer, and might answer negatively, is whether text-masked render agreement actually identifies correct readings. Everything else depends on it, so it is tested first.

Baselines

Frozen model. The naive reference is the frozen base model with greedy decoding, plus chain-of-thought prompting. We use SmolVLM2-2.2B so that our numbers sit next to Efficient Test-Time Scaling for Small Vision-Language Models. We add one larger open model. Chain-of-thought is included because MATH-Vision reports that it sometimes hurts multimodal models.

TTAug and TTAdapt. The strongest existing test-time method is TTAug/TTAdapt from the same paper, run with their perturbation set and $N=16$, the point where they report returns plateau. That paper reports a +6.5% mean improvement for TTAdapt and +4.1% for TTAug across nine benchmarks that include ChartQA. No per-benchmark ChartQA number was found on Lacuna, so we will reproduce it ourselves with matched backbone, adapter rank, step count and wall-clock budget per query. Beating it means a higher mean on the ChartQA human split and on Chart101 at equal or lower per-query latency. It also means a clear margin on the adversarial-title and added-grid-line conditions, where we predict agreement-based pseudolabels will reinforce errors.

Offline synthetic training. The strongest offline competitor is training on synthesized multi-step rationales, as in Synthesize Step-by-Step. That paper reports 53.9% on ChartQA human-written questions, up from a 38.2% prior best, 72.6% relaxed accuracy on ChartQA overall, and 92.89% on PlotQA. We cannot match their generation budget. Instead, we fine-tune our backbone on LAMENDA-style data at several corpus sizes, with and without style augmentation. We then plot accuracy against total generation-plus-training compute, placing our method on the same curve using its per-query cost multiplied by the test-set size.

Gradient-free self-consistency. Self-consistency with majority voting is run at matched sample count.

Ablations isolating the key ingredient. Four ablations each remove one piece. The first replaces render-verified tables with majority-vote tables (agreement instead of execution). The second removes text masking from the agreement function. The third drops the gradient step and answers $q$ by running a program over $T^\ast$. The fourth drops the anchor term on re-rendered images.

Why not just…?

Why not just answer from the render-verified table with a program, and skip gradient updates? This is the most serious threat, and it may win on purely table-answerable questions. The ablation that removes the gradient step settles it. We expect adaptation to help in two cases. One is questions that need visual relations not captured in a table, such as trends and comparisons of shapes. The other is online streams, where weights learned on earlier charts improve readings on later ones, even when no reading passes the check. If the program-only variant matches the full method in both modes, we will report that the renderer check, not test-time training, is the contribution.

Why not just use agreement-based TTAdapt, which already exists? Agreement measures confidence, not correctness. Plot Twist reports accuracy as high as 79% when the title contains the answer, falling to the 30% range when the title is removed or misleading. That pattern is exactly the one under which sampled answers agree on a wrong value. The adversarial-title and grid-line conditions of Chart101 test this directly.

Why not just use a frontier model as an external verifier? Limits and Gains of Test-Time Scaling shows that external verification helps: Claude on MMMU rises from 32.4% to 43.8% with Best-of-N and an external verifier. But each query then pays for a call to the verifier, and frontier models share the same blind spot. Gemini Pro Vision is about 10% less accurate on elementary chart questions than on complex ChartQA questions for the same images (Plot Twist). We include a frontier-verifier Best-of-N run on a subset as a reference, with its cost reported.

Why not just generate offline data with heavy style augmentation? It may cover common plotting libraries, and the compute-matched curve described in the Baselines section will show where it wins. The case for test-time adaptation is unseen styles. We will hold out one rendering library and marker family entirely from offline augmentation and test on it.

Proposed Experiments

Check 1: does render agreement work? This check comes first and can stop the project. On the 150 synthetic Chart101 plots, where the true tables are known, we sample readings and measure how well text-masked agreement separates correct from incorrect tables (AUROC). We compare this against answer-agreement frequency. If render agreement does not clearly beat agreement frequency, and especially if it fails under adversarial titles, the core claim is false.

Main evaluation. Next we evaluate on the ChartQA human split, PlotQA, and the Chart101 real-world plots and elementary questions. We report accuracy and collective accuracy, meaning all elementary questions about one chart answered correctly, a metric the Plot Twist authors report often falls below 25%. Chart101's stress conditions (title removed, misleading title, added grid lines, changed markers) form a shift suite. The Plot Twist authors report that PaLI-3 drops from 29.5% to 21.3% when grid lines are added. We measure whether our method shrinks such drops relative to TTAdapt.

Online mode. We stream charts from a held-out plotting style and report accuracy as a function of stream position, against single-query reset.

Stretch evaluation. The function-plot portion of MathVista tests whether gains carry to multi-step mathematical questions. For reference, MathVista reports GPT-4V at 49.9% and open models at 20–26% on the full benchmark, though we will report our own subset numbers.

Cost. Throughout, we report per-query latency and memory against TTAdapt and total compute against the offline-synthesis curve.

The project succeeds if render-verified adaptation beats TTAdapt on the shift suite at matched latency, and if the program-only ablation does not close that gap.