Video recognition models increasingly replace bare class names like "opening" with richer sentences written by a large language model (LLM), for example "gripping an edge and rotating a hinged part away from the body." These sentences help models attend to motion instead of the objects in the scene. There is a separate task, zero-shot compositional action recognition, where a model sees verbs and objects during training but must recognize pairings it has never seen, such as "folding a napkin." That task has moved the other way. Its best methods try to keep what the model knows about verbs statistically independent of what it knows about objects. The two trends conflict in a way nobody has measured. An LLM asked to describe "slicing an apple" will write about juice, curved fruit, and knife grip. The description ties the verb to its training partner, which is exactly the shortcut compositional methods try to remove.
This note proposes a controlled study and a simple method built around one measurable quantity. We call it object leakage: how much the text used to represent a verb reveals which objects that verb was paired with during training. The claim is that richer label text helps on unseen verb–object pairs only to the extent that it is object-blind. Leakage should predict the size of the gap between seen and unseen pairs. Generating verb descriptions with the object deliberately masked out, and filtering them for residual leakage, should keep the benefits of rich descriptions without re-creating the shortcut.
Core Claim
The work builds on two families of papers. The label-expansion papers motivate it. Diving Deep into the Motion Representation of Video-Text Models uses GPT-4 motion descriptions as classifier weights and shows the largest gains when objects are masked out of the labels (47.80% on masked UCF-101, 13 points over ViFi-CLIP). CLAVER and TP-CLIP pair LLM-written prompts with lightweight temporal modules. All three evaluate on ordinary splits, where objects and verbs co-occur the same way in training and test. None tests held-out pairings.
The compositional papers constrain the design. C2C uses a statistical independence penalty (the Hilbert-Schmidt Independence Criterion, HSIC) to decouple verb and object visual features. It still represents both with short lexical strings. LogicCAR adds fuzzy-logic constraints over an LLM-generated object taxonomy, but its labels remain discrete nodes. Something-Else takes object-agnosticism to the extreme: it replaces objects with role tags and discards appearance entirely. That makes subtle deformations like tearing invisible.
Our contribution sits between these. We keep text rich enough to describe mechanics. We make it blind to objects in the same sense that Something-Else's role tags are blind, and we measure blindness directly rather than assuming it.
We define leakage for a verb $v$ as the accuracy of a linear probe that predicts, from the frozen text embedding of $v$'s description, which training objects co-occurred with $v$. The probe is compared against a probe run on the bare verb name. We call this quantity $L$, and the task set is the verb–object compositions of the Something-composition benchmark. The central prediction is that the relative gap (seen accuracy minus unseen accuracy, divided by seen accuracy) grows monotonically with $L$ when everything else is held fixed. This mirrors the finding in Unobserved Local Structures Make Compositional Generalization Hard that failures on novel combinations come from specific local junctions between primitives. Here, the suspected junction is the verb–object association baked into the label text.
Research Program
We would build three kinds of text representation and plug each into an otherwise identical compositional model.
The first is composition-level expansion. The LLM describes each full verb–object pair. This is the obvious way to bring label expansion into compositional recognition, and it is our main foil.
The second is naive verb expansion. The LLM describes each verb given its class name alone, in the style of Devaraj et al. We expect this to leak moderately, because the LLM fills in typical objects on its own.
The third is our method, object-blind verb expansion. The verb is presented in Something-Something's own template form ("Folding [something]"). The LLM is instructed to describe hand configuration, contact changes, trajectory, and the resulting state of a generic "something." The output is then filtered. Any sentence whose embedding lets the leakage probe recover a training partner above chance is regenerated or dropped. A second LLM pass flags physically implausible statements, which addresses the hallucination concern raised for label expansion.
Objects get the reverse treatment. Their descriptions list the states an object can take ("a book: flat, open with pages fanned, bent at the spine"). This targets what C2C calls component domain variation, the fact that an object's appearance depends on what is being done to it. It does so without naming any particular action.
The compositional model is C2C with a CLIP backbone. The only change is that verb and object classifier weights come from averaged embeddings of the chosen descriptions rather than from class names. Its HSIC penalty and dual-path scoring stay in place, so any difference is attributable to the text.
To test causality rather than correlation, we construct a leakage dose series. Starting from object-blind descriptions, we insert the $k$ most frequent training objects for each verb into its description, with $k$ ranging from 0 to 5. This yields matched texts that differ only in leakage.
Baselines
The naive reference point is C2C with CLIP embeddings of the bare verb and object names. C2C reports a harmonic mean of seen and unseen accuracy (HM) of 44.5% on Sth-com with CLIP integration (C2C). We will reproduce it with our own seeds before comparing anything, because every other condition differs from it only in text.
The strongest existing method is LogicCAR. On Sth-com it reports 27.0% AUC and 57.9% unseen accuracy, against 25.9% AUC and 55.1% unseen for C2C as reproduced in the same paper (LogicCAR). Because those figures use AUC and unseen accuracy rather than HM, we will report all three metrics for every run and compare each metric only with its like. LogicCAR is the strongest competitor on this question because it is the only published method that already brings LLM knowledge into compositional action recognition, through a taxonomy rather than descriptions. We will run it with its released configuration on the same backbone, frames, and epochs as our model. We will also run a combined variant that adds object-blind descriptions to LogicCAR, since the two ideas are complementary.
Beating the baselines means two things. First, a gain in unseen accuracy and HM over the reproduced C2C-CLIP baseline that exceeds seed variance across three seeds. Second, no loss in AUC relative to LogicCAR, together with a gain when descriptions are added to it.
The key ablation is the pair of composition-level expansion and naive verb expansion, with training budget and number of prompts per class matched to our method. If our method beats naive expansion only where naive expansion leaks, that isolates object-blindness as the active ingredient. The leakage dose series is the second ablation. Tuning budgets are identical across all conditions: the same learning-rate sweep, three seeds, and a fixed number of descriptions per class.
Why not just…?
Why not just use a larger vision-language model and class names? Scale narrows but does not close compositional gaps. On the Compositional Generalization Gap of In-Context Learning finds that the relative gap shrinks with parameter count while absolute performance on out-of-distribution structure stays poor. Devaraj et al. report AI models below 60% on a motion retrieval study that humans solve at nearly 98%. We will still run the bare-name baseline and our method at two CLIP sizes. If the larger backbone erases the benefit of object-blind text, that is a real negative result and we will report it.
Why not just expand each composition directly, since that gives the richest text? This is the most serious threat, because composition-level descriptions contain strictly more information. Our hypothesis says it should win on seen pairs and lose on unseen ones. If it instead matches or beats our method on unseen accuracy, the leakage hypothesis is wrong, and the dose series will show a flat curve. The experiment is built to reveal exactly that.
Why not just rely on C2C's independence penalty to strip whatever the text leaks? HSIC in C2C acts on visual features. Classifier weights derived from text sit outside the penalty. Leaked object information in the verb weights can therefore reward visual verb features that encode objects, even when the penalty is applied. As a direct test, we add an HSIC term between text-derived verb and object weights and check whether it recovers the gains of object-blind generation. If it does, the simpler regularizer is preferable, and we will say so.
Why not just augment with swapped verb–object pairs? Good-Enough Compositional Data Augmentation lifts SCAN from 0% to 87% but fails to improve on SCHOLAR. C2C already uses a visual analogue, CutMix "imagination," which is present in every one of our conditions. String swaps do nothing about the visual change in an object across actions, which is what our object-state descriptions target.
Proposed Experiments
The main benchmark is Something-composition: 161 verbs, 248 objects, and about 80,000 clips with disjoint seen and unseen pairs. We report seen accuracy, unseen accuracy, HM, and AUC. Something-Else serves as a second compositional split. There the reference is the STIN+I3D ensemble, which improves top-1 from 46.8% to 58.1% (Something-Else). Masked-object UCF-101 and HMDB-51, following Devaraj et al., check that object-blind text does not sacrifice ordinary zero-shot transfer.
Three checks are decisive.
The first is correlational. Across all text conditions and verbs, per-verb leakage $L$ should correlate positively with the per-verb relative gap. We will report this as a rank correlation and as an AUC for predicting which unseen compositions fail, in the spirit of the local-structure failure predictor.
The second is causal. In the dose series, unseen accuracy should fall as $k$ rises while seen accuracy stays flat or improves.
The third is practical. Object-blind descriptions should improve unseen accuracy and HM over the reproduced C2C-CLIP baseline and add to LogicCAR.
If the dose series is flat, the proposal is falsified: leakage in label text is harmless, and composition-level expansion is the better engineering choice. A falsification of that kind would itself tell the field how to combine these two lines of work.
References
- Diving Deep into the Motion Representation of Video-Text Models
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners
- Is Temporal Prompting All We Need For Limited Labeled Action Recognition?
- C2C: Component-to-Composition Learning for Zero-Shot Compositional Action Recognition
- Zero-shot Compositional Action Recognition with Neural Logic Constraints
- Something-Else: Compositional Action Recognition With Spatial-Temporal Interaction Networks
- Unobserved Local Structures Make Compositional Generalization Hard
- On the Compositional Generalization Gap of In-Context Learning
- Good-Enough Compositional Data Augmentation