Contrastive Evidence Grammars for Fine-Grained Action Recognition

Video models often confuse actions that look almost the same: mopping versus sweeping, opening versus closing a container, a windmill dunk versus an ordinary layup. The information that separates them is usually small and brief, such as the tool head at the moment of floor contact, or the direction of the hand at the instant it touches the lid. This proposal treats telling such actions apart as a multi-hop search. The model first has to find the moment that matters and then inspect the right detail at that moment. This mirrors text question answering, where a system must first resolve "the director of Inception" before it can look up a birthplace. We propose to force each search step to be contrastive. At every step the model must name a visual attribute on which the remaining candidate actions actually disagree, retrieve the video evidence for that attribute, and drop the candidates the evidence rules out. The decoder enforces this rule while the model writes. It is not just a requested output format.

Put in two sentences: build, at every step, a grammar from the current candidate set, so the model can only ask questions whose answers would separate those candidates. Then let the answers drive which frames are retrieved next, until one candidate remains. The bet is that the constrained decoding that structured-output research uses to guarantee syntax can instead guarantee that each reasoning step is discriminative. That property is what fine-grained action recognition lacks. Multimodal models tend to fall back on language priors and on whole-scene appearance, which do not separate these actions.

Core Claim

Grammar-constrained decoding is a technique that masks out, at each generation step, any token that would break a formal grammar. Its standard use is to make output parseable. SynCode attaches an incremental parser and token masks to the language model and removes all JSON syntax errors. The authors themselves note that such grammars cannot enforce semantic constraints. We claim that, in a closed-world disambiguation problem, part of the semantics can be compiled into the grammar. Once the candidate set is known, the set of useful next questions is finite and can be computed. Let $C_t$ be the surviving candidate actions after hop $t$. Let $\mathcal{A}$ be a set of observable attributes, such as contact state, motion direction, tool-head type, or body-part trajectory, and let $v(c,a)$ be the expected value of attribute $a$ under action $c$. The grammar at hop $t+1$ admits a query on attribute $a$ only if

$$\exists\, c, c' \in C_t:\; v(c,a) \neq v(c',a),$$

meaning the attribute must separate at least two surviving candidates. Candidate names, frame indices, and attribute values are likewise restricted to legal tokens.

The empirical claim has two parts. First, this contrastive grammar should beat the same multi-hop protocol run without constraints, and should also beat generic schema-constrained output. The gain should be largest on benchmarks built to defeat static-appearance shortcuts. Second, the gain should come from the discriminative restriction, not from syntactic validity alone. The ablations below are designed to separate these two sources.

Where this sits relative to prior work

Three video systems motivate the proposal, and each leaves a specific gap.

Video-STAR breaks actions into body-part sub-motions and calls a pose tool and a text retrieval module for action definitions. It encourages correct formatting through a soft reinforcement-learning reward. Its retrieval is a single lookup of definitions, its formatting is not guaranteed, and nothing requires a sub-motion check to separate competing candidates. We keep its tools but make retrieval iterative over the video and make discrimination a hard rule.

ActAlign has a language model write ordered sub-action scripts for each class and aligns them to frames with dynamic time warping, a method for matching two sequences that may run at different speeds. This happens in one pass, without looking at the video, and global alignment is hurt by filler frames. Our hops instead retrieve local frame windows conditioned on earlier findings.

LLaVAction adds a supervised action token, mines hard distractors, and re-ranks a specialist's top-$k$ candidates. It also documents visual neglect, the tendency of multimodal models to rely on language priors over visual detail. Its output structure is a flat verb–noun pair, and its accuracy is capped by the recall of the specialist's top-$k$ set. ActionAtlas supplies the core test of static-appearance bias, because in that benchmark candidate classes share the same scene.

On the text side, two papers supply the hop structure. IRCoT uses each reasoning sentence as the next retrieval query. GRAT searches over sub-questions with a tree search guided by step-level rewards. Both emit free-form intermediate queries, and the research file names unstructured sub-question formulation as a source of chains drifting off track. We borrow the "the result of hop $t$ determines the query of hop $t+1$" pattern and replace free text with a constrained query language.

On the structured-output side, the work constrains our design. The cognitive-multitasking hypothesis attributed to Haikang Deng in the file holds that imposing rigid formats degrades reasoning. Grammar-Aligned Decoding shows that naive masking distorts the model's output distribution. We therefore constrain only the query, argument, and elimination fields, and leave a free-text rationale field unconstrained. We treat the hypothesis as something to test, not to assume.

Research Program

The system runs a loop over a video and a candidate set. Candidates come from the benchmark's multiple-choice options, or from a recall-oriented specialist shortlist with a large $k$.

Offline, a language model builds an attribute table $v(c,a)$ for each candidate. Where available, it draws on retrieved definitions, in the manner of Video-STAR's retrieval module. Online, each hop works in three stages. The model emits a constrained query that names an attribute admissible under the equation above, together with a localization request. Tools then execute the request. Frame retrieval uses SigLIP similarity to an attribute-specific text prompt, following ActAlign's embedding space. Pose estimation follows Video-STAR. A crop-and-zoom operation handles small details such as hand regions. The model then reads the retrieved frames and emits a constrained observation and elimination record, and $C_{t+1}$ is computed deterministically from that record.

The first hop is typically localization ("when does hand–object contact occur?"). This is the video analogue of resolving the bridging entity. Later hops inspect discriminating attributes inside the localized window.

The grammar is regenerated after every hop as a small context-free grammar and compiled with an incremental masking engine of the SynCode type. Because the vocabulary of legal arguments is tiny, the memory cost of the mask store, flagged in the file as 1–2 GB for full programming languages, should be negligible here.

Two failure modes need explicit handling. The first is a wrong attribute table, which is the fragility of generated sub-actions noted for ActAlign. To cover this, the grammar includes a "contradiction" production. When observed evidence fits no surviving candidate, this production reopens eliminated candidates or requests a table revision. The second is the recall ceiling of stage-one filtering. We measure how often the gold label is absent from the shortlist and how often the reopen production recovers it.

The system runs training-free on a Qwen2-VL-class 7B backbone. As a second phase, we optionally fine-tune with GRPO, the group-relative policy optimization method Video-STAR uses, with rewards only for answer correctness and hop count. Format rewards become unnecessary because format is guaranteed.

Baselines

The naive reference point is the same 7B multimodal model answering the multiple-choice question directly from uniformly sampled frames, with no tools and no hops. A second reference is the same model asked for chain-of-thought reasoning before answering.

The strongest training-free competitor on static-appearance-controlled data is ActAlign, which reports 30.4% top-1 accuracy on ActionAtlas with 0.9B parameters. The same paper reports Qwen2-VL-7B at about 30.24%. We will rerun ActAlign on our splits with its released configuration, and we will also give it our attribute tables as its sub-action scripts, so that both methods share the same language-model knowledge.

The strongest trained competitor on hard-distractor data is LLaVAction. It reports 74.1% accuracy on EPIC-KITCHENS-100-MQA (Hard), against 52.2% for GPT-4o. Our system will use LLaVAction-7B itself as its perception backbone and as the source of its stage-one shortlist. Any gain then comes from the hop protocol rather than from a different model. Beating it means exceeding 74.1% on the same Hard split with the same backbone.

Video-STAR is the closest tool-using competitor. It reports 92.1% on HMDB-51; no reported result found on Lacuna for Video-STAR on ActionAtlas or EPIC-KITCHENS-100-MQA. We will run it with the same tools and the same frame budget.

Three ablations isolate the key ingredient, holding tools, backbone, frame budget, and maximum hop count fixed. The first is an IRCoT-style loop whose queries are free text. The second is the identical protocol with a fixed JSON schema enforced by constrained decoding but without the contrastive admissibility rule. The third is the contrastive rule stated in the prompt but not enforced. All variants receive the same prompt-tuning budget on a shared development split.

Why not just…?

Why not just use standard JSON-mode constrained decoding on a Video-STAR-style agent? Constrained decoding would remove malformed tool calls, which is the failure Video-STAR's soft reward leaves open. However, a schema that admits any attribute lets the model keep asking questions that do not separate the candidates, such as "is a person holding a handle?" for mopping versus sweeping. The second ablation in the Baselines section is this alternative exactly. If it matches the full method, our contribution reduces to syntactic hygiene, and we will report it as such.

Why not just prompt the model to compare the candidates? This is the most serious threat, because strong models may follow a contrastive instruction without needing enforcement. The evidence against it is indirect. LLaVAction documents that multimodal models default to language associations when faced with fine-grained distractors, and IRCoT-style free-form queries are known to drift. The third ablation settles the question. We will also log the fraction of non-discriminating queries the unconstrained model produces. If that fraction is low and accuracy is equal, we drop the enforcement claim and keep the hop protocol.

Why not just fine-tune with more hard negatives, as LLaVAction does? Fine-tuning works within a domain. The file, however, reports that specialists trained on egocentric kitchens transfer poorly to general benchmarks. Our method needs no video training, so the decisive comparison is cross-domain. We will train or adopt LLaVAction-style models on EPIC-KITCHENS and evaluate both systems on ActionAtlas.

Doesn't constraining output hurt reasoning, as the cognitive-multitasking hypothesis predicts? It may. We therefore compare full-output constraint against constraint applied only to the query and elimination fields. Measuring the accuracy gap between these two settings is itself a test of the hypothesis in a multimodal setting.

Proposed Experiments

The main evaluations are ActionAtlas, which tests static-appearance-controlled domain actions, and EPIC-KITCHENS-100-MQA (Hard), which tests specialist-mined distractors. HMDB-51 serves as a sanity check where ambiguity is lower and gains should be small. Metrics are top-1 accuracy, malformed-call rate, mean hops, wall-clock latency, recovery rate of the reopen production, and an audit of hop faithfulness. For the audit, annotators check on a 300-clip sample whether each hop's cited frames actually show the claimed attribute.

The decisive checks are three. The full method must beat the unenforced-contrastive ablation by a margin outside bootstrap confidence intervals on ActionAtlas. The gain must concentrate on candidate pairs that share a scene, which we identify by measuring how well a single-frame classifier separates them. And on EPIC-KITCHENS-100-MQA (Hard), the method must exceed LLaVAction's 74.1% when built on LLaVAction's own backbone.

A negative result on the first check would show that discriminative questioning can be learned from instruction alone. That would also be worth knowing for the agentic structured-output literature.