If a robot learns to "open a door" and "pick up a cup," we expect it to intuitively understand how to "open a cupboard" or "pick up a plate." However, standard deep learning models often fail this basic test of logic. They tend to treat "opening a door" as a single, monolithic visual pattern. When the door is replaced by a cupboard, the model’s recognition of the "opening" action often vanishes because it has entangled the motion of the verb with the appearance of the noun.
This failure is known as the compositional gap. Recent research focuses on bridging this gap by explicitly decoupling action representations from object identities, allowing models to generalize to novel combinations they have never seen during training.
The Problem of Visual Entanglement
In traditional action recognition, a model might identify a "playing piano" clip by simply detecting the piano. As Chinmaya Devaraj, Cornelia Fermuller, and Yiannis Aloimonos point out in Diving Deep into the Motion Representation of Video-Text Models, many state-of-the-art models are "motion-blind," relying on static object cues as proxies for actions.
This entanglement creates a bottleneck for generalization. If a model learns "cutting" only in the context of "apples," it fails to recognize "cutting a cake" because the visual features of the cake do not match the "apple-cutting" template it memorized. Junhyuk Oh and colleagues observed this in reinforcement learning, noting that agents fail subtasks involving novel pairings because they cannot disentangle the underlying relationship between an action and its target Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning.
Learning Actions through Placeholders
To force models to learn the "essence" of an action, researchers have begun stripping away the objects during training. A powerful technique involves using generic placeholders in text descriptions.
In Toward Open-Set Human Object Interaction Detection, Mingrui Wu and their team introduced Disentangled HOI Detection (DHD). Instead of training a model to recognize "riding a horse," they use prompts like "a person doing something" or "a person interacting with an object." By matching visual features to these generic action prompts, the model learns to identify the verb independently of the noun. This allowed their system to generalize from 600 known interactions to over 17,000 novel ones, such as "riding a dolphin."
Similarly, Chinmaya Devaraj et al. replaced specific nouns with placeholders in video-text training to ensure the model focuses on physical dynamics—like the way fingers move—rather than the presence of a piano.
Interaction Centric Architectures
Beyond changing the training data, new architectures are being designed to prioritize how things move over what they are.
- Motion-Guided Attention: Tae Soo Kim 0001 and Gregory D. Hager developed a dual-pathway model where object detections "teach" a motion pathway where to look Motion Guided Attention Fusion to Recognize Interactions from Videos. This explicitly separates static appearance from dynamic movement.
- Object-Region Transformers: Roei Herzig and Anna Rohrbach introduced ORViT, which allows transformer patches to attend to specific object trajectories Object-Region Video Transformers. This helps the model decouple the action from the visual context of the training objects.
- Relative Dynamics: Debaditya Roy and Basura Fernando argue for an "interaction-centric" approach that prioritizes the relative motion between hands and objects Modelling Spatio-Temporal Interactions for Compositional Action Recognition. By focusing on trajectories rather than pixel-level appearance, their model can identify interactions even when the objects belong to entirely new categories.
Why Decoupling Works
The intuition behind this research direction is that actions are functional, while objects are categorical. The physical "how" of an action (the trajectory, the force, the relative change in position) is often consistent across different objects.
By using structured knowledge like And-Or Graphs to "hallucinate" diverse action sequences (Tianshui Chen et al., Neural Task Planning with And-Or Graph Representations) or using lightweight motion filters to isolate temporal shifts (Myunggi Lee et al., Motion Feature Network), these researchers are moving away from "action-blind" detection.
The result is a more robust form of AI that doesn't just recognize a scene, but understands the underlying mechanics of interaction. When we decouple the "what" from the "how," we move closer to systems that can reason about the world with the same compositional flexibility as humans.