Structured video captioners write their descriptions as fixed fields: one for the main object, one for the background, one for camera movement, and so on. They still do two things badly. First, nothing guarantees the output is valid. A dropped key or bracket breaks the parser downstream. Second, they compress the video the same way for every field. A model that has thrown away 80–90% of its visual tokens to save compute describes a red jacket well enough, but it fails on a slow pan, because the evidence for camera motion lives in fine differences between frames that uniform compression erases.
This note proposes to fix both problems with one mechanism. A grammar automaton is a small state machine that tracks which part of the schema the model is currently writing and masks out tokens that would make the output invalid. We would run such an automaton during decoding and use its current state for two jobs. The first is the usual one: masking invalid tokens so the output always parses. The second job is new: choosing which pool of visual tokens the language model may attend to. Inside the camera field, the decoder sees tokens that keep motion detail: low spatial resolution, many frames. Inside main_object, it sees tokens that keep spatial detail: high resolution, few frames. In short, the schema is no longer just an output format. It becomes a switch that routes visual evidence, and the grammar that guarantees validity is the same thing that picks that evidence.
Core Claim
At a matched visual-token budget per decoding step, routing visual tokens by schema field should beat uniform token compression on the fields that need a different kind of evidence, above all camera movement and temporal transitions. Grammar-constrained decoding would also give 100% schema validity, at little latency cost.
The claim is falsifiable in a specific way. If the camera and background scores do not improve over a uniform-compression model with the same number of attended tokens, the routing idea is wrong, whatever happens to validity.
Where This Sits
Each piece of prior work plays a different role here. Three captioning systems motivate the gap. AuroraCap captions videos in five categories (short, main object, background, camera, detailed). It merges 80–90% of visual tokens uniformly, and it reports that it still struggles with pans, zooms and subtle transitions. TimeChat-Captioner writes a six-field audio-visual script. It enforces that structure only through supervised fine-tuning and reinforcement learning with the GRPO algorithm. InstanceCap gets per-instance fields by building an external pipeline of detectors, the SAM2 segmentation model, and blurring around each instance before prompting a multimodal model.
That last design is the closest precedent for "show the model different evidence for different fields." However, it runs as many separate prompts through an offline pipeline, and schema compliance is left to post-processing. Our proposal moves the evidence selection inside a single decoding pass and ties it to a formal grammar.
Two decoding papers supply the method and constrain the design. Mitigating Bias in Locally Constrained Decoding via Tractable Proposals shows two things. Plain token masking is myopic and can walk into dead ends. A lookahead check against a tensorized automaton, which they call GCD, reaches 100% constraint satisfaction on the xLAM function-calling benchmark at about 1.1x overhead. We reuse GCD as the validity layer. Their probabilistic variant costs 3.11x to 10.2x and needs a distillation step, so we avoid it.
Draft-Conditioned Constrained Decoding (DCCD) finds a "projection tax": forcing syntax from the first token distorts reasoning. Their fix is to write a free-form draft first and then a constrained pass conditioned on the draft. With it, a 1B model on GSM8K rises from 15.2% to 39.0% strict accuracy. That result is a direct threat to any single-pass constrained captioner, so we measure the tax for video rather than assume it away.
Finally, ReVisiT steers each decoding step using the model's own vision tokens to reduce hallucination. It shows that decoding-time use of visual evidence is cheap, at 1–3% overhead. It knows nothing about schemas, though, and it works on single images, not multi-field video scripts.
Research Program
The build has three parts.
The first part prepares the visual tokens. We encode the video once into two or three token pools, all cached together.
- An appearance pool is merged spatially, in AuroraCap's style, but keeps full resolution on a few keyframes.
- A motion pool is heavily downsampled in space but dense in time, and adds frame-difference tokens.
- For audio-visual schemas, an audio pool comes from the backbone's audio encoder.
The second part is the schema automaton, compiled from the JSON schema. Each field is tagged with a routing rule that says which pools it may see and with what token budget. The rule is written as an attention mask over the cached visual tokens, $M(s)$, where $s$ is the automaton's current field. Because the mask depends only on the field, the visual key-value cache is computed once, and switching fields costs nothing beyond the mask.
The third part is fine-tuning with routing turned on. Field boundaries are known exactly in the training JSON, so the model learns to use each field's pool. We would fine-tune AuroraCap-7B on VDC-style training data and TimeChat-Captioner's released backbone on its schema. During training the routing masks are applied and grammar masking is off. At inference, both are on.
Before building anything, we answer an analysis question: does a format tax exist for video captioning at all? Take the same model and decode the same fields three ways: unconstrained, with plain token masking, and DCCD-style (free draft, then constrained rewrite with the video still visible). Then compare per-field scores. The structured-captioning literature suggests schemas help, and the DCCD result suggests forced syntax hurts. The per-field comparison tells us which effect dominates, and on which fields.
If the tax is large, the routing decoder gets a draft stage. If it is negligible, single-pass constrained decoding is justified, and that finding is itself worth reporting.
Baselines
The naive reference is the base model decoding unconstrained, with its output parsed by a standard JSON parser. We report two numbers for it: its content score and its parse-failure rate.
The strongest existing method on audio-visual scripting is TimeChat-Captioner-7B. It reports a SodaM score of 35.0 on OmniDCBench, above Gemini-2.5-Pro at 33.7 (TimeChat-Captioner). On OmniDCBench, beating it means exceeding 35.0 SodaM while also reaching 100% schema validity, with the same backbone and fine-tuning data plus routing. On VDC, the reference is AuroraCap-7B with its uniform token merging. The file reports that it leads open-source models on VDCScore, but no reported result found on Lacuna for its exact VDCScore value, so we will re-run it ourselves with the released model. There, beating it means a higher camera-category VDCScore at an equal number of attended visual tokens per step.
A validity baseline applies GCD on top of each of these models without routing. This separates the effect of guaranteed parsing from the effect of routed evidence.
A multimodal DCCD baseline produces a free draft and then a constrained rewrite. We run it with visual access and with a matched total token budget. This tests whether drafting, rather than routing, explains any gain.
A per-field prompting baseline queries the model once per field. Each query uses that field's high-resolution token pool and a field-specific question, in the spirit of InstanceCap's per-instance prompting. This baseline gets K times the prefill compute, where K is the number of fields, and we report that cost explicitly.
The key ablation keeps everything identical but swaps the routing masks. In one version, every field sees a uniform pool of the same size. In another, fields get randomly permuted pools. If neither ablation loses to the proposed method on camera and background fields, the field-to-evidence assignment is doing nothing.
All systems share the training data, the fine-tuning steps, and a fixed decoding hyperparameter sweep of eight configurations each on a held-out split.
Why not just…?
Why not just rely on SFT and GRPO to teach the format, as TimeChat-Captioner does? Training makes valid output likely but not certain, and schema changes require retraining. More importantly, training does not address the evidence problem: a model that learned the format perfectly still sees uniformly compressed video in its camera field. The naive-reference and validity baselines measure the parse-failure rate of the trained model directly. If that rate is already near zero, the validity half of this proposal is minor, and the contribution rests on routing alone. We would say so.
Why not just mask invalid tokens with an off-the-shelf constrained decoder? Plain masking can get stuck in states with no valid completion within the length budget (Mitigating Bias in Locally Constrained Decoding via Tractable Proposals). It can also impose a projection tax on content (DCCD). We include it in the format-tax study. In any case, masking only fixes syntax and cannot help the camera field.
Why not just caption freely and let a text LLM restructure the output? A text-only restructurer can only split what the captioner already wrote. If the free caption confuses a pan with subject motion, as the structured-captioning papers argue unstructured captions do (MiraData, InstanceCap), the error cannot be recovered from text. The multimodal DCCD baseline is the strong version of this idea, since its rewrite pass still sees the video. If that baseline matches routing, the honest conclusion is that drafting is enough.
Why not just prompt the model separately for each field with the right tokens? This is the most serious threat. Per-field prompting can plausibly match routing on quality, because it gives each field exactly the evidence routing would. Our claim then becomes an efficiency claim: routing gives the same field-specific evidence in one pass with one shared cache, at roughly 1/K the prefill cost. It may also give better cross-field consistency, because later fields can see earlier ones. We test this by checking whether the object named in main_object agrees with the one referenced in events. If per-field prompting wins on quality and consistency at acceptable cost, we will report routing as the cheaper approximation, not as a quality gain.
Why not just scale the model? A 7B specialist already beats Gemini-2.5-Pro on OmniDCBench, and the camera-motion weakness comes from compression, which a larger decoder does not undo.
Proposed Experiments
The main benchmarks are VDC, scored with VDCScore per category, and OmniDCBench, scored with SodaM overall and per field. On both we also report schema validity rate and wall-clock latency.
The decisive check has two parts. We would need a camera-category VDCScore gain over the matched-budget uniform ablation, with no loss on the main-object field. We would also need the same pattern on OmniDCBench's camera-state and shot-editing fields.
Secondary checks follow from the sections above:
- the format-tax study, which reports per-field score differences between unconstrained, masked, and draft-conditioned decoding;
- latency, where the target is under 1.2x the unconstrained model, consistent with GCD's reported ~1.1x;
- downstream question answering on DailyOmni, where TimeChat-Captioner's captions lift accuracy from 13.4 to 52.8 and we check whether routed scripts raise that further.
A final transfer test decodes MiraData-style six-field captions with a model trained on VDC's five fields, mapping the new fields onto existing pools. This asks whether routing by field type carries over to new schemas without retraining.
References
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
- TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
- InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption
- Mitigating Bias in Locally Constrained Decoding via Tractable Proposals
- Draft-Conditioned Constrained Decoding for Structured Generation in LLMs
- Revisit What You See: Disclose Language Prior in Vision Tokens for Efficient Guided Decoding of LVLMs
- MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions