Knowing When the Blackboard Is Finished: Evidence-Sufficiency Gating for Streaming Mathematical Video

Picture a model watching a live math lecture. Someone asks it a question whose answer depends on a derivation that the lecturer has not finished writing yet. The model has two ways to fail. It can answer too early, guessing from the problem statement before the key step appears, or it can wait so long that the answer is useless. Streaming video models today mostly fail the first way. On forward-looking questions they answer prematurely almost every time (PhoStream). Offline, the same model families also cannot reason reliably over diagrams and equations even when they see them clearly (MATH-Vision).

We propose to treat this as the problem that text-stream summarization studied a decade ago: trading how fast an update arrives against how complete it is. That field scored systems with latency-discounted gain (TREC 2013 Temporal Summarization; A multi-level system for sequential update summarization). We apply the same idea to mathematical reasoning over live video. The method has two parts. The first is a running symbolic transcript of the board, which we call the ledger: equations, labeled quantities and diagram facts, rewritten as text or code only when the board changes at the level of symbols. The second is an emission rule that answers only once the answer computed from the ledger stops changing as new evidence arrives. The claim is that waiting for evidence to settle, and reasoning over symbols rather than raw pixels, beats both relevance-triggered answering and waiting until the video ends. Performance is measured on a latency-discounted accuracy curve.

Core Claim

In a streaming math video, each question has an evidence-sufficiency time $t^*$: the earliest moment at which everything needed to answer it has been shown or said. We make three claims.

First, current streaming systems emit most of their answers before $t^*$. Their correct answers are therefore largely lucky guesses from the problem framing. This extends to symbolic reasoning the early-response finding of PhoStream and the title-shortcut finding of Plot Twist.

Second, a gate that fires when the ledger-derived answer is stable across successive ledger updates lands close to $t^*$ much more often than two alternatives: a relevance-and-density gate of the kind used in QueryStream, or a gate on the model's own stated confidence.

Third, keeping the board as symbols matters for accuracy and not just timing. The reason is that lightweight image-embedding filters can throw away the thin strokes and small annotations that reasoning depends on.

What is new is not a benchmark of streaming failure; PhoStream and OVO-Bench already supply that. Nor is it symbolic de-rendering, which Chart-based Reasoning already supplies. What is new is a timing mechanism tied to when the evidence is sufficient, an evaluation that credits answers only after that point, and an incremental version of de-rendering whose cost is paid once per board change rather than once per query.

Where the Prior Work Stands

Three strands of work motivate the proposal.

The text-stream summarization tracks formalized the tension between speed and completeness. They also documented both failure modes: aggressive filtering produces tunnel vision (BJUT at TREC 2014), and recall-oriented systems flood the user (University of Lugano at TREC 2015). In live deployment, median latency ranged from about one second to more than a hundred hours (Overview of the TREC 2018 Real-Time Summarization Track). These systems were purely textual and extractive. We borrow their scoring logic, not their methods.

The multimodal reasoning literature locates the bottleneck after perception. The largest error category for GPT-4V on MATH-Vision, at 42.2%, is reasoning over correctly perceived content (MATH-Vision). Models still fail when diagrams are replaced by human-written text (GeoSQA). They recognize components of exam diagrams but cannot chain operations over them (GAOKAO-MM).

The streaming video work motivates the timing problem. PhoStream shows impatience in Forward questions. It also shows that audio makes impatience worse, because speech runs ahead of visual confirmation. PhoStream measures this failure but proposes no fix, and its questions are about mobile assistance rather than derivations. VideoMathQA supplies exactly the derivation-style content we need, but evaluates it offline, with the whole video available.

Other work constrains the design. MATH-Vision reports that multimodal chain-of-thought (asking the model to write out intermediate reasoning steps) can lower accuracy. We therefore cannot assume that longer reasoning at emission time helps, and we test that directly. OVO-Bench reports roughly 4 seconds to process 64 frames, with steep growth in frame count. This rules out any design that re-reads the whole history at every tick, and is the reason the ledger is updated incrementally. QueryStream prunes over 70% of visual tokens using OpenCLIP relevance to the query plus novelty against a smoothed history, and triggers answers when relevance and density gates open. It is the closest mechanism to ours, and it differs in two ways. Its trigger measures how relevant and how new the incoming frames are, not whether the answer has converged. Its pruner can discard symbol-level detail that image embeddings do not resolve.

Two lines of work supply methods we reuse. Chart-to-table de-rendering followed by program-of-thought execution raised a 5B model from 77.3% to 81.3% on ChartQA (Chart-based Reasoning). Tool- and template-generated executable rationales are a related source of training data (Synthesize Step-by-Step). We apply this kind of de-rendering to board regions as they change. Budget-aware compute methods, which skip layers under a stated budget (AdaLLaVA) or route to expensive tools (BOLT), operate on static inputs. We treat them as compatible add-ons, not competitors, and keep them out of the core comparison.

Research Program

Building the evaluation. We first build Streaming VideoMathQA. For each of the 420 VideoMathQA questions, two annotators mark $t^*$, the earliest timestamp at which every needed visual and spoken fact has appeared. Disagreements go to a third annotator, and we report inter-annotator agreement in seconds. The video is then fed at one frame per second with its time-aligned transcript. The question is posed at a fixed early time, and the system may emit one answer at any later moment, or none.

The score adapts expected latency gain from the temporal summarization tracks. Let $c_q$ be 1 if the emitted answer to question $q$ is correct and 0 otherwise. Let $t_q$ be the wall-clock emission time, which includes compute time. Let $\tau$ be a delay scale. The per-question score is

$$ s_q = c_q \cdot \mathbb{1}[t_q \ge t^*_q] \cdot \exp\!\big(-(t_q - t^*_q)/\tau\big). $$

We report the mean of $s_q$ across a sweep of $\tau$ values, giving an accuracy-versus-delay curve. Alongside it we report the early-response rate, meaning the fraction of emissions made before $t^*$, and plain accuracy with no timing constraint. Correct answers emitted before $t^*$ earn nothing in $s_q$. They are reported separately as guess accuracy, because they reveal shortcut use in the sense of Plot Twist.

The ledger. The ledger maintainer runs a cheap board-change detector on text-region crops. When a region changes at the level of symbols, and only then, it runs a de-renderer that writes the new content as LaTeX, a table, or a short list of diagram facts. Examples of diagram facts are "AB = 5" and "angle C is a right angle". Spoken statements that assert quantities are appended from the transcript, tagged as spoken, so the gate can tell spoken claims from visually confirmed ones. That distinction targets the audio-induced impatience PhoStream observed. The de-renderer is the backbone model prompted or lightly fine-tuned on rendered equation and diagram crops in the style of Chart-based Reasoning.

The gate. After each ledger update, the backbone computes a candidate answer from the ledger plus the current frame. Where the answer is numeric or symbolic it does so through a short executable program, and otherwise through direct answering. The gate emits when the same answer appears across $k$ consecutive ledger updates, or across $\Delta$ seconds without any update, and no spoken-only fact is load-bearing in the program. Both $k$ and $\Delta$ are tuned on a held-out split.

Analysis. We then ask whether a stable answer is a good proxy for sufficient evidence. We measure the distribution of $t_q - t^*_q$. We check whether errors made after $t^*$ trace to ledger transcription mistakes or to reasoning over a correct ledger. Following MATH-Vision, this separates perception errors from reasoning errors, now under time pressure.

Baselines

All streaming systems share one backbone, Qwen2.5-VL-7B, because QueryStream was built on it. They share the same 1 fps input, the same transcript, the same GPU type, the same maximum output tokens per emission, and an equal hyperparameter tuning budget on the same held-out split. Wall-clock compute is counted into $t_q$ for every system.

The first reference point is emit at question time, answering the moment the question is posed. It shows how much is achievable by guessing, and is expected to post a large early-response rate.

The second is wait until the video ends, then answer with chain-of-thought over the full video and subtitles. This is VideoMathQA's own offline protocol. It maximizes completeness and sets the ceiling for accuracy without timing. The strongest offline number the file reports on this benchmark is 44.8% accuracy for GPT-4o-mini with chain-of-thought and subtitles in the multi-binary setting (VideoMathQA). We rerun our backbone under the same protocol rather than compare across models. We found no reported result on Lacuna for any streaming or latency-constrained evaluation on VideoMathQA, so every streaming number here will be new.

The strongest existing streaming method is QueryStream: its pruning plus its dual relevance-and-density answer trigger, run training-free on the same backbone (QueryStream). It reports 75.32 on StreamingBench (StreamingBench) and a 2.2-point gain on the long subset of Video-MME. On OVO-Bench forward responding it reports no absolute number, only a qualitative improvement over density-only triggers. It is the right competitor because it is the only method in the file that decides when to answer in a stream. Beating it means a higher area under the latency-discounted curve across the $\tau$ sweep with non-overlapping bootstrap intervals, and a lower early-response rate. We also require that the gain holds when QueryStream is given our ledger as extra context. That condition tests whether the advantage comes from the gate and not only from better input.

Four ablations isolate our ingredients. The first is our stability gate over raw frames without the ledger. The second is the ledger with QueryStream's trigger. The third is the ledger with a confidence-threshold gate. The fourth removes the spoken-versus-visual tagging.

Why not just…?

Why not just transcribe speech and subtitles and let a text model do the reasoning? This is a real threat, because lecturers narrate much of what they write. GeoSQA shows that even perfect text renderings of diagrams do not rescue reasoning. That finding cuts against any approach, ours included, so it motivates the program-execution step more than the ledger itself. The sharper evidence is from PhoStream: speech precedes visual confirmation and increases premature answers. We run a transcript-only system with the same stability gate. If it matches the full system on VideoMathQA questions that are mostly narrated, we will report that the visual ledger matters only for questions whose key evidence is diagram-only. We will stratify results by that property, annotated alongside $t^*$.

Why not just answer whenever the model says it is confident? Model confidence mixes certainty about the prior with certainty about the evidence, which is exactly what produces early guessing. A 97.89% early-response rate in PhoStream is hard to square with calibrated self-assessment. Still, a well-tuned threshold on sampled self-consistency could approach a stability rule. The third ablation, the ledger with a confidence gate, tests this directly. If it ties our gate, we will present the ledger, not the stability gate, as the contribution.

Why not just wait until the end and think step by step? It gives up latency entirely, and the $\tau$ sweep will show where the extra accuracy stops paying for the delay. MATH-Vision also reports that multimodal chain-of-thought can lower accuracy, so waiting plus long reasoning may not even win on accuracy without timing. VideoMathQA's finding that long videos degrade performance suggests that reasoning over a compact ledger may beat reasoning over an hour of frames even offline. We test this by running the ledger at the end of the video.

Why not just use QueryStream's gate, which already waits for relevant content? Its gate fires when relevant, novel frames accumulate. A derivation can be highly relevant long before it is complete, so its trigger should systematically fire before $t^*$ on multi-step problems. Its OpenCLIP pruning may also drop symbol-level detail. The second ablation separates these two effects.

Proposed Experiments

The decisive test is on Streaming VideoMathQA. We compare the latency-discounted curve and the early-response rate of the full system against QueryStream and the two naive reference points, stratified by reasoning type (Problem Focused, Concept Transfer, Deep Instructional Comprehension) and by video length.

The prediction is specific. The largest gains should come on Deep Instructional Comprehension and long videos, where evidence arrives late and in pieces. The gains should be small on Problem Focused questions, whose evidence arrives early. If gains are uniform, the timing story is wrong, and any improvement is coming from the ledger's representation alone.

A second check measures the median $|t_q - t^*_q|$ for each gate. This tests whether answer stability actually tracks evidence sufficiency. It is the claim most likely to fail, for example if models settle confidently on a wrong answer early.

For generality, the gate without the symbolic ledger is run on PhoStream Forward questions and OVO-Bench forward active responding. The reference points there are Doubao-Seed-1.6's Forward score of 44.26 and Qwen3-Omni's 97.89% early-response rate, both from PhoStream. This tells us whether stability gating helps outside mathematics, where no symbols exist to de-render.

Finally, we report the wall-clock cost of ledger maintenance per board change. The aim is to show that incremental de-rendering stays within the roughly per-second budget of the input stream, rather than adding the multi-second delays that made heavy quality filtering self-defeating in BJUT at TREC 2017.