Zero-Shot Compositional Generalization in Visual Interaction Detection

20 concepts19 papers1 proposal

When humans look at a scene, we can easily understand novel combinations of familiar concepts. If you know what a "tiger" looks like and you know what the action "riding" entails, you can instantly recognize someone "riding a tiger," even if you have never seen that specific scenario before.

For machine perception models, however, this task is notoriously difficult. In visual relationship detection (VRD), scene graph generation (SGG), and human-object interaction (HOI) detection, models are asked to parse scenes into structured triplets: (subject, predicate, object). Because the real world contains a massive variety of entities and actions, the combination of these elements suffers from a combinatorial explosion. As noted by Wenbo Li et al. in their work on concurrent activity detection, it is practically impossible for any training dataset to provide exhaustive coverage of every potential interaction pair.

This creates a severe compositional generalization gap. Standard deep learning models tend to entangle the appearance of the entities with the specific actions or spatial layouts seen during training. When confronted with an unseen composition of a known verb and a known object—a zero-shot triplet—they fail.

To solve this, recent research focuses on a shared goal: decoupling the constituent parts of an interaction so they can be flexibly recombined, allowing models to generalize to the long tail of unseen real-world interactions.


Geometric and Algebraic Disentanglement

One of the earliest and most intuitive ways to achieve compositionality is to enforce geometric or algebraic constraints on how subjects, predicates, and objects relate to one another in a latent space. If the network is forced to learn a structured mathematical relationship between these components, it cannot easily overfit to specific training pairs.

A foundational example of this approach is VTransE, proposed by Hanwang Zhang et al. in Visual Translation Embedding Network for Visual Relation Detection. VTransE maps visual features into a low-dimensional relation space where the predicate acts as a simple vector translation between the subject and the object:

$$\text{subject} + \text{predicate} \approx \text{object}$$

By regularizing the network so that an action like "riding" always corresponds to a consistent directional shift—whether the subject is riding a bicycle, a horse, or a tiger—the model decouples the predicate's meaning from the specific entities involved.

   Subject Space                 Relation Space                  Object Space
+-----------------+            +-----------------+            +-----------------+
|  [Person Feat]  | ---------> |  [Person Vector]|            |                 |
+-----------------+            +-----------------+            |                 |
                                        |                     |                 |
                                        | + [Ride Vector]     |                 |
                                        v                     |                 |
                               +-----------------+            +-----------------+
                               |  [Target Vector]| ---------> |  [Tiger Feat]   |
                               +-----------------+            +-----------------+

Instead of a simple vector translation, Yuting Qiang et al. addressed this combinatorial space in Tensor Composition Net for Visual Relationship Prediction by modeling interactions as a low-rank 3-way tensor. By utilizing Tucker decomposition, their Tensor Composition Net (TCN) breaks down the massive relationship space into core tensors and factor matrices. Unseen relationships can then be predicted simply by multiplying the learned latent factors of the novel subject, predicate, and object combinations.

More recently, researchers have explored higher-dimensional structural equivariance. In Navigating the Unseen: Zero-shot Scene Graph Generation via Capsule-Based Equivariant Features, Wenhuan Huang et al. leveraged Capsule Networks to model relationships as equivariant transformations. By treating unseen interactions as geometric rotations or transformations of seen ones within a high-dimensional feature space, the model preserves the underlying spatial and semantic properties across entirely different object categories.


Decoupling Localization from Interaction Recognition

Standard object detectors are highly optimized to output a bounding box and a class label simultaneously. However, when these architectures are extended to interaction detection, they often entangle where an object is and what it is with how it is being interacted with. For example, if a model only sees the verb "eating" associated with a specific spatial layout (e.g., a person sitting at a table with a plate), it will fail to recognize "eating an apple while walking."

To prevent this spatial and semantic entanglement, a prominent strategy is to completely separate object localization from interaction recognition.

Standard Entangled Pipeline:
[Image] ---> [Joint Detector & Interaction Predictor] ---> Overfits to seen layouts

Decoupled Pipeline:
             +---> [Detector-Agnostic Localization] ----+
[Image] ----|                                           |---> [Independent Interaction Matcher]
             +---> [Semantic / Spatial Feature Extraction] -+

Sho Inayoshi et al. tackled this in Bounding-Box Channels for Visual Relationship Detection by introducing Bounding-box Channels (BBC). Their architecture encodes spatial layouts into dedicated feature channels that remain spatially aligned with image features, preventing the loss of spatial context during feature fusion and allowing the model to decouple learned spatial patterns from specific semantic categories.

In the domain of zero-shot HOI detection, Yunyao Mao et al. demonstrated in CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection that returning to a strict two-stage paradigm—where object localization is entirely isolated from interaction identification—prevents the model from overfitting to specific training layouts. This allows the interaction module to focus purely on the behavioral cues of the action.

Building on this decoupled philosophy, Shiyu Xuan et al. proposed DA-HOI, a framework that isolates interaction recognition so completely that it becomes detector-agnostic. By using a deterministic matching strategy, it can be plugged into any pre-trained object detector without retraining, ensuring that the model's ability to locate objects is never compromised by the rarity of the interaction.

This decoupling also solves a critical bottleneck: proposing candidate interactions. As Mingrui Wu et al. pointed out in End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation, traditional models fail to even propose a candidate interaction if the verb is unfamiliar. By introducing an action-agnostic interaction discovery module, they ensure the system can locate potential interactions before attempting to classify them.

Taking a different structural approach to this search problem, Jiyan Wu reframed the task in Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection as a human-centric query. Instead of detecting all objects and exhaustively matching them, the model anchors its search on detected humans and uses them as queries to scan the scene for interacting entities across multiple scales, bypassing the combinatorial matching bottleneck entirely.


Fabricating the Unseen and Leveraging Open-World Knowledge

If a specific triplet is entirely absent from the training data, how can a model learn what its visual representation looks like? Modern frameworks solve this using two main strategies: data fabrication (hallucinating features or images) and knowledge distillation from open-world foundation models.

Hallucinating Synthetic Features

Instead of waiting to encounter rare interactions in the real world, models can synthetically compose them in the feature space. Baosheng Yu et al. introduced this concept via Fabricated Compositional Learning (FCL) in Detecting Human-Object Interaction via Fabricated Compositional Learning. FCL uses an "object fabricator" to generate synthetic object representations, which are then combined with real verb features to create artificial training samples for rare or unseen triplets.

[Real Verb Features] ------+
                           v
                     [Composition Mechanism] ---> [Synthetic Training Triplet]
                           ^
[Object Fabricator] -------+
  (Generates Fake Objects)

With the advent of generative AI, this fabrication can now happen at the image level. In Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models, Liulei Li et al. repurposed text-to-image diffusion models to generate synthetic training data. By learning specialized "relation embeddings" to represent actions, their framework, DIFFUSIONHOI, extracts spatially-aware visual features directly from generated images of zero-shot interactions.

Distilling Open-World Common Sense

An alternative to generating new data is leveraging large-scale vision-language models (VLMs) like CLIP, which have already seen millions of diverse text-image compositions during web-scale pre-training.

To adapt these insights for localized interaction detection, Ting Lei et al. developed CMMP in Exploring Conditional Multi-modal Prompts for Zero-Shot HOI Detection. CMMP decouples vision and language prompts within CLIP and injects spatial priors, allowing the model to map localized visual regions to open-vocabulary interaction descriptions.

Rather than relying on rigid classification heads, some frameworks reframe interaction detection entirely as a sequence-to-sequence problem. In Open-Category Human-Object Interaction Pre-training via Language Modeling Framework, Boshen Xu et al. introduced OpenCat, which treats HOI triplets as sequences of text tokens. By pre-training on massive amounts of weakly supervised image-caption data, the model learns to generate interaction descriptions autoregressively, bypassing the constraints of a closed vocabulary. A similar philosophy drives the SCoRD framework, which uses an auto-regressive transformer augmented with ungrounded text captions to predict all relations associated with a specific subject of interest.

Constraining the Open World

While open-world models provide incredible generalization, they also introduce a new problem: they can predict visually plausible but logically absurd or physically impossible triplets.

To counter this, Jiankai Li et al. introduced the T-CAR framework in Zero-shot Scene Graph Generation via Triplet Calibration and Reduction. T-CAR uses a calibration loss to mitigate bias toward frequent triplets and applies a space reduction technique based on Positive-Unlabeled (PU) learning to filter out unrealistic compositions, forcing the model to focus its capacity on plausible but rare relationships.

Complementing this statistical filtering, Liulei Li et al. introduced LogicHOI in Neural-Logic Human-Object Interaction Detection, which blends Transformer architectures with symbolic logic. By embedding common-sense rules directly into the neural network's loss function, they ensure that zero-shot predictions conform to logical real-world constraints.


Summary

Zero-shot compositional generalization works because it shifts the paradigm from memorizing triplets to understanding components and rules. By enforcing algebraic structures (like vector translations or tensor decompositions), decoupling localization from interaction features to prevent spatial overfitting, and leveraging the open-world common sense of vision-language models, these architectures can successfully interpret novel interactions in unpredictable real-world environments.

Key Foundational Work

Visual Translation Embedding Network for Visual Relation Detection (2017) — contributed 1 concept elements to this direction.
Joint Learning of Object and Action Detectors (2017) — contributed 1 concept elements to this direction.
Bounding-Box Channels for Visual Relationship Detection (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_generalization_zero_shot_i_caa23e81,
  title = {compositional generalization (zero-shot interaction recognition)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-zero-shot-interaction-recognition/rdr_caa23e81fd784c92ae65ad3a41895989},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_generalization_zero_shot_i_caa23e81_v1,
  title = {compositional generalization (zero-shot interaction recognition)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-zero-shot-interaction-recognition/rdr_caa23e81fd784c92ae65ad3a41895989/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake