Novel Human-Object Interaction Detection via Adversarial Domain Generalization

Yuhang Song, Wenbo Li, Lei Zhang, Jianwei Yang, Emre Kiciman, Hamid Palangi, Jianfeng Gao, C.-c. Jay Kuo, Pengchuan ZhangarXiv 2020 · OpenReview

Abstract

Human-Object Interaction (HOI) detection is a critical computer vision task that involves identifying triplets of <subject, predicate, object>. A major challenge in this field is the large compositional space of objects and predicates, leading to a lack of sufficient training data for all possible combinations. Most existing methods rely heavily on object priors and struggle to generalize to unseen scenarios. This paper proposes a unified framework of Adversarial Domain Generalization (ADG) to learn object-invariant features for predicate prediction. By treating different object categories as distinct domains, the model is forced to isolate the visual representation of the action itself. Experimental results on a new split of the HICO-DET dataset and the UnRel dataset show performance improvements of up to 50% and 125%, respectively, in detecting novel HOIs.

The Challenge: The Long-Tail Problem

The primary hurdle in HOI detection is the long-tail/combinatorial problem. While a dataset might contain thousands of images of a person "riding a horse," it may contain zero examples of a person "riding a dog." Standard models often fall into a "frequency prior" trap, where they associate the visual features of the object (the horse) with the action (riding). Consequently, when presented with an unusual or unseen combination, the model fails because it hasn't learned to recognize the "verb" independently of the "noun."

Figure 1: Distribution of HOI triplets in the HICO-DET dataset, illustrating the extreme imbalance and long-tail nature of interaction data.

Methodology: Adversarial Domain Generalization

To address this, the researchers introduced the Adversarial Domain Generalization (ADG) framework. The core strategy is to treat each object category (e.g., bicycle, horse, car) as a separate domain. The goal is to extract features for the predicate (the action) that are consistent across all these domains.

The architecture utilizes an adversarial setup consisting of:

  1. A Feature Extractor: This module extracts visual features from the "union box" containing both the human and the object.
  2. A Discriminator: This component attempts to identify the object category based solely on the extracted features.
  3. Adversarial Training: The feature extractor is trained to "confuse" the discriminator. If the discriminator cannot determine whether the object is a horse or a bicycle, it indicates that the feature extractor has successfully isolated the action (e.g., "riding") from the specific object involved.

The authors explored several variations, including Conditional ADG (CADG), which ensures features are similar across objects only when the action is identical, acknowledging that "holding" a phone looks visually different from "holding" a surfboard.

Key Results and Insights

The proposed framework was evaluated on a custom split of the HICO-DET dataset (focused on unseen triplets) and the UnRel dataset (focused on unusual relations).

  • Performance Gains: The ADG framework increased performance by up to 50% on unseen HICO-DET categories and a staggering 125% on the UnRel dataset compared to baseline models.
  • Visualizing the "Verb": Using Grad-CAM, the authors observed that their model focuses on the interaction space between the human and the object, whereas baseline models often fixate solely on the object itself.
  • Generalization to the Unusual: The model successfully detected "unusual" interactions that were entirely absent from the training data, such as a person riding a dog.
Figure 5: Examples of HOI detection. The model successfully predicts the "ride (dog)" interaction, a novel combination not seen during training, by focusing on object-invariant action features.

Conclusion

By shifting the focus from memorizing object-predicate pairings to learning object-invariant action features, the ADG framework significantly advances the state-of-the-art in novel HOI detection. While the system remains dependent on the accuracy of the underlying object detector, the adversarial approach provides a robust "plug-and-play" module for improving the generalization capabilities of existing HOI architectures. Future work may involve incorporating semantic knowledge graphs to further bridge the gap between seen and unseen interactions.

Summary updated

Abstract

We study in this paper the problem of novel human-object interaction (HOI) detection, aiming at improving the generalization ability of the model to unseen scenarios. The challenge mainly stems from the large compositional space of objects and predicates, which leads to the lack of sufficient training data for all the object-predicate combinations. As a result, most existing HOI methods heavily rely on object priors and can hardly generalize to unseen combinations. To tackle this problem, we propose a unified framework of adversarial domain generalization to learn object-invariant features for predicate prediction. To measure the performance improvement, we create a new split of the HICO-DET dataset, where the HOIs in the test set are all unseen triplet categories in the training set. Our experiments show that the proposed framework significantly increases the performance by up to 50% on the new split of HICO-DET dataset and up to 125% on the UnRel dataset for auxiliary evaluation in detecting novel HOIs.
BibTeX
@misc{song2020novel,
  author = {Yuhang Song and Wenbo Li and Lei Zhang and Jianwei Yang and Emre Kiciman and Hamid Palangi and Jianfeng Gao and C.-C. Jay Kuo and Pengchuan Zhang},
  title = {{Novel Human-Object Interaction Detection via Adversarial Domain Generalization}},
  year = {2020},
  eprint = {2005.11406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.11406},
}
Figure