Temporal Derendering: Giving Black-Box Video Models a Verified State Log of Animated Charts

Frontier multimodal models can describe a single frame of an animated chart accurately. They still fail to say how the chart changed: which bar overtook which, how many times a line crossed a threshold, or whether the animation slowed down. We propose to train a small, open model on procedurally rendered animated charts, where the true data value of every mark at every timestamp is known because we generated it. The model learns to turn a video into a timestamped state log, a compact text record of what each data series looked like and when key events happened. That log is pasted into the prompt of a frozen proprietary model accessed only through an API. The question is whether this external perceptual record breaks the ceiling that limits text-only steering of black-box models, or whether the black box fails anyway.

This matters for two reasons. Proprietary models remain weak on genuinely temporal questions: GPT-4o scores 37.7% on TOMATO against 95.2% for humans, and most models score below 37.5% on DVBench's animation-pacing questions. The existing tool for adapting such models without weight access, small "advisor" models that write per-input advice, was shown to give no gain on MATH-500. That result suggests advice cannot supply a capability the black box lacks. If the temporal deficit is mainly perceptual, meaning the model cannot track states across frames, then supplying a verified state log should close much of it. If the deficit is mainly in reasoning, it should not. This proposal tests which of the two holds.

Core Claim

A compact model trained on rendered animated charts, with supervision that is exact by construction, can produce a temporal state log that substantially raises a frozen black-box model's accuracy on dynamic-chart questions. It should do so most on pacing, cross-frame aggregation and counting. The gain must come from modeling time, not from reading each frame separately. Concretely, the log should beat the obvious alternative of running a static chart-to-table converter on sampled frames and concatenating the tables. Its benefit should also disappear when the input video's frames are shuffled.

This differs from the three pieces of closest work.

How to Train Your Advisor trains a text-only advisor with Group Relative Policy Optimization (GRPO, a reinforcement learning method that compares groups of sampled outputs), using the black box's task score as reward. It never sees pixels, and its MATH-500 null result is the constraint we probe. Our scaffold carries perceptual content, not advice.

Synthesize Step-by-Step and Chart-based Reasoning supply the method, converting charts back into tables and distilling program-like rationales. Both are single-frame and both fine-tune the answering model with gradient access. The second also shows that table-only rationales lose color and style information. Our renderer keeps that information because it knows the visual encodings.

DVBench motivates the work by measuring the failure. It is evaluation only, and it notes that automatic generation of verifiable temporal-chart questions is unsolved.

Research Program

Generator. We build a procedural renderer for animated charts covering bar-chart races, growing lines, morphing scatter plots, stacked-area transitions and sorted-order changes. Each animation uses controlled timing curves, including non-linear easing, pauses and speed changes, so that pacing is a learnable, labeled quantity. We vary plotting style, grid lines, color palettes and titles. Plot Twist showed that grid lines and informative titles strongly move model accuracy, so this variation serves both as training augmentation and as a test of whether the student reads the marks rather than the text. Deceptive or blank titles are included deliberately.

Questions and verification. Because the generator stores every mark's value, color and position at each timestamp, every question is written as a program over the ground-truth state. Examples are "when did series A first exceed B," "how many rank changes occurred," and "during which interval was the animation fastest." Answers are obtained by running that program. A teacher model writes step-by-step rationales from the true state log. We keep only rationales whose embedded program reproduces the program's answer. This is the same execute-to-verify filtering that succeeds for SQL in SQLForge and for summaries in Referee. Here the checker is the renderer itself, which addresses the verifier bottleneck for temporal tasks.

Student. A roughly 7B open video-language model is fine-tuned to output the state log from raw frames. The log records values per series at key times, detected events with timestamps, and the visual encoding of each series. The student also learns to output an optional short rationale. After a first round, we follow Let GPT be a Math Tutor and generate new animations aimed at the student's measured failure categories.

Coupling to the black box. At test time the log is placed in the black-box prompt alongside the frames. A second, optional stage then trains the student with GRPO using the black box's correctness as reward, using the advisor-models code. The aim is to learn what to include and how to compress the log. This stage is an ablation, not the core claim. API cost and reward noise make it expensive, so we keep it only if it adds measurably over the supervised log.

Baselines

The naive reference is each black-box model answering from frames alone with its default sampling. DVBench reports Gemini-3.1-Pro at 77.91% average accuracy, and most models below 37.5% on the pacing layer. TOMATO reports GPT-4o at 37.7% and Qwen2-VL-72B at 37.9%. We re-run these models ourselves under our frame budget, since our numbers must be compared on identical inputs.

The strongest simple competitor is per-frame static derendering. A DePlot-style chart-to-table converter, the component used in Synthesize Step-by-Step, is run on the same sampled frames, and the timestamped tables are concatenated into the black-box prompt. No reported result was found on Lacuna for this pipeline on DVBench or TOMATO. It is the comparison that isolates our key ingredient: whether learning across frames, which covers identity tracking, events between sampled frames and timing, beats reading frames independently. For a fair comparison, the converter is fine-tuned on the same synthetic charts, treated as single frames, with matched compute. It also receives the same frame count and the same prompt-length budget.

The strongest existing black-box adaptation method is the text advisor of How to Train Your Advisor. Because it is text-only, we give it frame captions from an open captioner so that it can be run at all. We train it with the same number of black-box API calls as our GRPO stage. No reported result was found on Lacuna for advisor models on any video or chart benchmark. Its reported text gains are +27.4% on RuleArena Taxes, rising from 67.2% to 85.6%, and none on MATH-500. Those figures frame what "breaking the ceiling" would look like.

We also test a static optimized prompt found by edit-based search as in GrIPS, plus zero-shot chain-of-thought. Finally, we evaluate the student answering directly, without the black box.

Beating the baselines means two things. First, on the DVBench pacing and animation questions, our log must exceed the per-frame derendering baseline by a margin larger than the confidence interval across seeds and resamples. With only 1,000 DVBench questions, we report bootstrap intervals and supplement them with a held-out synthetic test set of real-styled animations. Second, our log must not lower accuracy on narrative questions.

Why not just…?

Why not just derender each frame with an existing static chart model and concatenate the tables? This is the real threat, and it may capture most of the gain on aggregation questions where values are readable in every sampled frame. It should fail where the answer depends on what happens between samples or on timing, such as pacing, rank swaps during transitions, and series identity when colors re-sort. Those are exactly DVBench's weakest categories. The matched comparison in Baselines settles the question. If per-frame derendering matches us, we report that temporal modeling is unnecessary and that the contribution is the verified synthetic data alone.

Why not just fine-tune an open video model on the synthetic data and skip the black box? That student is one of our baselines. It may well win on synthetic tests. DVBench, however, mixes chart reading with narrative and audio-visual alignment, where large proprietary models are strongest. The hybrid should keep the black box's narrative strength while fixing perception. If the student alone matches the hybrid, black-box coupling is unjustified, and we will say so.

Why not just feed more frames or ask the model to think step by step? DVBench reports that denser sampling does not improve accuracy steadily, and TOMATO finds a plateau after 8 frames. MATH-Vision finds that chain-of-thought sometimes lowers multimodal accuracy. We nonetheless run both at our frame budget, because these findings come from other models and settings.

Why not just optimize a single prompt? The advisor paper reports static optimizers at 40–60% reward against 85–100% for per-instance steering on FSPO. A fixed prompt also cannot add information that is absent from the black box's perception, which is our hypothesis about the deficit. The GrIPS baseline tests this directly.

Proposed Experiments

The main evaluation runs DVBench by dimension with two to three proprietary black boxes, for example Gemini-3.1-Pro and GPT-5.4, under the four conditions: none, per-frame tables, our log, and our log after GRPO. ChartQA serves as a check that static reading is not harmed by our training. TOMATO's Shape & Trend, Velocity & Frequency and Action Count categories form an out-of-domain probe. These involve natural video rather than charts, so we expect little transfer and treat any gain as a bonus, not a claim.

Three checks are decisive.

The shuffle control borrows TOMATO's frame-order sensitivity idea. We shuffle the frames given to our student and confirm that the log's benefit collapses, whereas the per-frame baseline should be largely unaffected. If our gain survives shuffling, the model is not using time.

The ceiling test compares advice with a log. On questions where the captioned text advisor gives no gain, does the state log help? If so, the advisor ceiling on temporal tasks is perceptual. If the log also fails there, the black box's temporal reasoning itself is the bottleneck. That is a useful negative result.

The oracle-log upper bound feeds the black box the true rendered state log on synthetic test videos. This separates errors made by our student from errors the black box makes even when perception is perfect.