Compositional Action Representations for Zero Shot Generalization

135 concepts94 papers

When a human encounters a novel instruction like "slice a dragonfruit," they do not need to see a demonstration to understand what it means. Even if they have never seen a dragonfruit before, they can instantly combine their existing understanding of the action "slicing" with their visual recognition of the new object.

Traditional deep learning models for activity recognition struggle with this task. If a model is trained on monolithic labels like "slicing bread" and "peeling an apple," it treats each action-object pair as an isolated category. Consequently, it fails to recognize "slicing an apple" or "peeling bread" because it has never seen those exact pixel combinations before.

To overcome this limitation, a growing body of research focuses on compositional action representation. By decoupling actions from their targets, models can generalize to an open-ended world of activities without requiring exhaustive training data for every possible permutation.

The Monolithic Label Trap and Combinatorial Explosion

In conventional computer vision, human activity recognition is often framed as a standard classification problem: a neural network maps an entire video or image to a single categorical label. As noted by Haonan Yu et al. in A Compositional Framework for Grounding Language Inference, Generation, and Acquisition in Video, this approach completely ignores the compositional structure of human behavior and language.

Treating action-object pairs as monolithic entities creates two severe bottlenecks:

  1. The Long-Tail Problem: Real-world human-object interactions follow a heavy long-tail distribution. As the benchmark dataset HICO demonstrates, collecting enough joint training data for rare or unusual interactions is practically impossible.
  2. Quadratic Parameter Explosion: If an environment contains $V$ verbs and $N$ nouns, a monolithic model must learn $V \times N$ distinct classes. As Zhekun Luo et al. argue in Disentangled Action Recognition with Knowledge Bases, this quadratic scaling prevents models from expanding to open-vocabulary settings.

Furthermore, monolithic classification creates a high-stakes prediction problem. In an uncertainty-aware analysis of action anticipation, the authors of Uncertainty-aware Action Decoupling Transformer for Action Anticipation point out that when a model treats an action as a single atomic label, a failure to recognize either the object or the motion results in a total prediction error. The model cannot leverage the fact that it might be highly confident about the motion even if the object is ambiguous.

The Mechanism of Action-Object Decoupling

The core insight of compositional action modeling is to force the neural network architecture to factorize its representations into independent components—typically verbs (actions/affordances) and nouns (objects/targets).

Monolithic Approach:
[ Video Frame ] ---> [ Deep Network ] ---> Class: "Washing a Tomato" (Fails on unseen pairs)

Compositional Approach:
                      /---> [ Verb Stream ] ---> Verb: "Washing" ----\
[ Video Frame ] ----<                                                 >---> Combined: "Washing a Tomato"
                      \---> [ Noun Stream ] ---> Noun: "Tomato" -----/

By separating the visual representation of an action from the specific object involved, the model's parameter space scales linearly ($V + N$) rather than quadratically.

To achieve this separation, models are often designed with dual-stream architectures or factorized text prompts. For example, Mingrui Wu introduces a framework in Towards Open-Set Human Object Interaction Detection that trains models using generic placeholders in text prompts (e.g., "a person doing something"). By matching human visual features to these object-agnostic prompts, the model learns to isolate the visual signature of the verb itself, preventing the identity of the object from obscuring the representation of the interaction.

Once decoupled, these features can be recombined dynamically. Yingan Liu et al. demonstrate in Compositional action recognition with multi-view feature fusion that cross-attention mechanisms can effectively fuse these independent verb and noun representations. Similarly, in Cross-Task Weakly Supervised Learning From Instructional Videos., Dimitri Zhukov shows that the final classification score for a complex activity can be computed by mathematically combining the outputs of separate component classifiers. This allows a model to leverage data across completely different tasks; it can improve its "pouring" classifier by watching someone pour milk, and then immediately apply that knowledge to recognize someone pouring oil.

Granular Primitives: Moving Beyond Verbs and Nouns

While factorizing activities into verbs and nouns provides an immediate boost to generalization, high-level verbs like "cutting" or "cooking" can still be highly variable. True compositional generalization requires breaking actions down even further into atomic physical primitives—the fundamental movements of limbs, joints, and postures.

The advantage of this granular approach is structural: the variety of possible human body part arrangements is significantly smaller than the total variety of high-level human activities. As the authors of Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection explain, representing complex interactions as combinations of atomic body part movements allows for zero-shot recognition via a recombinant strategy.

Several frameworks implement this hierarchical decomposition:

Recombination in Generative and Decision-Making Agents

The utility of compositional action representations extends far beyond classification. By mastering factorized representations, generative models can "imagine" new scenarios, and reinforcement learning agents can navigate massive action spaces efficiently.

Zero-Shot Video Synthesis

In generative modeling, decoupling motion patterns from visual identities allows networks to synthesize videos of interactions they have never seen. Megha Nawhal et al. demonstrate this in Generating Videos of Zero-Shot Compositions of Actions and Objects using HOI-GAN. By processing action and object word embeddings as separate inputs and utilizing spatio-temporal scene graphs, the model can successfully synthesize a video of "washing a tomato" after training only on "washing an aubergine" and "putting a tomato."

In text-based interactive fiction games (like Zork), an agent faces an open-ended natural language action space. Generating full command strings directly is incredibly inefficient. To solve this, Dongwon Ryu et al. implement a template-object structure in A Minimal Approach for Natural Language Action Space in Text-based Games. The model separately selects an action template (e.g., open [OBJECT]) and a target object argument. This structural compositionality allows the agent to drastically reduce its search space and immediately apply the logic of "opening" from a door to a chest without explicit retraining (Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning).

Robotics and Task Planning

For cognitive robotics, compositionality is essential for executing multi-step tasks in dynamic environments. In Neural Task Planning with And-Or Graph Representations, Tianshui Chen et al. decouple primitive actions from target objects during task step predictions. This allows a robot to generalize its planning policy to novel combinations—such as grasping a tool it has never interacted with in that specific context before—because the action representation remains invariant to the object's identity.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_generalization_composition_6ca259e5,
  title = {compositional generalization (compositional action representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-compositional-action-representation/rdr_6ca259e5164f42c49eb8c4b3acb276e8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_generalization_composition_6ca259e5_v1,
  title = {compositional generalization (compositional action representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-compositional-action-representation/rdr_6ca259e5164f42c49eb8c4b3acb276e8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake