Agglomerative Transformer for Human-Object Interaction Detection

Danyang Tu, Wei Sun, Guangtao Zhai, Wei Shen

Introduction: The Integrality Problem in HOI Detection

Human-Object Interaction (HOI) detection is a complex computer vision task that goes beyond simple object recognition. It requires a model to understand the semantic relationship between a person and an object—for instance, determining whether a person is sitting on, carrying, or looking at a chair. To achieve this, models must capture fine-grained "instance-level cues," such as body pose and gaze direction.

Historically, single-stage Transformer-based models have struggled with what researchers call the "integrality" problem. Traditional models use sparse "instance queries" to locate objects. However, these queries tend to be "lazy"; they often focus only on the most distinguishable parts of an object (like a person's head or feet) rather than the entire instance. This sparse focus is insufficient for tasks like pose estimation, which require information from the whole body.

Comparison between sparse attention heatmaps (left) and integral instance masks (right) produced by AGER.

The AGER Framework: Agglomerative Clustering for Instance Tokens

To solve the integrality problem, the authors introduce AGER (Agglomerative Transformer). Instead of relying on a fixed set of learned queries, AGER treats the creation of instance representations as a text-guided dynamic clustering process.

Agglomerative Clustering and Textual Guidance

AGER starts with raw image patches and progressively "agglomerates" or clusters related patches into a smaller set of instance tokens within the encoder. To ensure these clusters correspond to meaningful semantic objects rather than random background textures, the model utilizes a frozen CLIP text encoder. During training, the visual instance tokens are encouraged to align with text embeddings of their respective labels (e.g., "a photo of a person").

Instance-Level Cue Extraction

Because these instance tokens are "integral"—meaning they represent the entire object region—they serve as rich information carriers. AGER passes these tokens through lightweight Multi-Layer Perceptrons (MLPs) to simultaneously predict secondary cues, including 17-point human poses and spatial bounding boxes. This architectural choice allows the model to extract fine-grained details without the need for the slow multi-stage pipelines (like cropping and re-processing) used in older methods.

The AGER pipeline for extracting and aggregating instance-level cues using dynamic instance tokens.

Key Results and Performance

AGER was evaluated on the HICO-Det and V-COCO benchmarks, achieving a new state-of-the-art performance of 36.75 mAP on the HICO-Det "Default" test set.

Beyond accuracy, AGER demonstrates significant improvements in efficiency and precision:

  • Efficiency: AGER is 36% faster in terms of frames per second (FPS) and uses 8.5% fewer GFLOPs than the standard QPIC baseline. This efficiency stems from the clustering process, which reduces the total number of tokens the decoder must process in later stages.
  • Precision Boost: Compared to prior query-based methods like GEN-VLKT, AGER’s instance tokens provide a 3x improvement in the precision of cue extraction (pose and gaze), validating the "integrality" of its tokens.
  • Long-Tail Robustness: The inclusion of extra cues significantly improved the model's ability to recognize rare interactions, showing up to a 10.1% performance gain in categories with scarce training data.

Limitations and Future Outlook

While AGER sets a new benchmark, it faces challenges with very small or heavily occluded objects where the spatial resolution for clustering is limited. Additionally, the current interaction decoder uses a "brute force" pairwise enumeration, checking every possible human-object pair. Future research may focus on more sophisticated filtering mechanisms to prune non-interactive pairs earlier in the pipeline, further enhancing speed in crowded environments.

Summary updated

Abstract

We propose an agglomerative Transformer (AGER) that enables Transformer-based human-object interaction (HOI) detectors to flexibly exploit extra instance-level cues in a single-stage and end-to-end manner for the first time. AGER acquires instance tokens by dynamically clustering patch tokens and aligning cluster centers to instances with textual guidance, thus enjoying two benefits: 1) Integrality: each instance token is encouraged to contain all discriminative feature regions of an instance, which demonstrates a significant improvement in the extraction of different instance-level cues and subsequently leads to a new state-of-the-art performance of HOI detection with 36.75 mAP on HICO-Det. 2) Efficiency: the dynamical clustering mechanism allows AGER to generate instance tokens jointly with the feature learning of the Transformer encoder, eliminating the need of an additional object detector or instance decoder in prior methods, thus allowing the extraction of desirable extra cues for HOI detection in a single-stage and end-to-end pipeline. Concretely, AGER reduces GFLOPs by 8.5% and improves FPS by 36%, even compared to a vanilla DETR-like pipeline without extra cue extraction. The code will be available at https://github.com/six6607/AGER.git.
BibTeX
@inproceedings{tu2023agglomerative,
  author = {Danyang Tu and Wei Sun and Guangtao Zhai and Wei Shen},
  title = {{Agglomerative Transformer for Human-Object Interaction Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure