Improving Human-Object Interaction Detection via Virtual Image Learning

Shuman Fang, Shuai Liu, Jie Li, Guannan Jiang, Xianming Lin, Rongrong Ji

Abstract

Human-Object Interaction (HOI) detection is frequently hindered by the "long-tail" distribution of real-world datasets, where rare interactions lack sufficient visual representation for effective model training. This paper discusses Virtual Image Learning (VIL), a framework designed to augment training data with high-quality synthetic images. By utilizing a Multiple Steps Image Creation (MUSIC) pipeline and an Adaptive Matching-and-Filtering (AMF) system, VIL effectively addresses the scarcity of rare HOI samples. Experimental results on HICO-Det and V-COCO benchmarks demonstrate that VIL significantly improves performance, particularly for rare interaction categories, setting new state-of-the-art benchmarks.

Introduction

Human-Object Interaction (HOI) detection aims to identify triplets of <human, verb, object> within an image, providing a granular understanding of visual scenes. While deep learning has advanced this field, the "long-tail" problem remains a significant obstacle. Common interactions, such as "person using a laptop," are abundant in datasets like HICO-Det, whereas rare interactions, such as "person holding a sheep," are critically underrepresented. This imbalance causes models to overfit to frequent actions and fail to generalize to rare ones.

To overcome this, researchers have introduced Virtual Image Learning (VIL), a training strategy that leverages generative AI to create synthetic training data for these rare categories.

The VIL Framework

The VIL framework consists of two primary components: the MUSIC pipeline for data generation and the AMF module for robust model training.

MUSIC: Multiple Steps Image Creation

Generating useful training data from text-to-image models like Stable Diffusion requires more than simple prompting. The MUSIC pipeline introduces a structured approach to ensure synthetic images are both diverse and semantically accurate:

  1. Text Refinement: Instead of generic prompts, MUSIC adds specific characterizations (e.g., "an old man") and environmental contexts (e.g., "in a corral") to enhance realism.
  2. Multi-Filtering: To prevent the inclusion of "hallucinated" or low-quality images, MUSIC applies three filters:
  3. Scene Similarity: Ensures the generated image matches the visual style of real-world datasets.
  4. Instance Existence: Uses an object detector to verify that the required human and object instances are present.
  5. Interactiveness: Employs CLIP to confirm that the human and object are actually performing the specified interaction.

AMF: Adaptive Matching-and-Filtering

Synthetic images often contain "noise" or missing labels for secondary interactions. The AMF module employs a Teacher-Student framework to manage this. A "Teacher" model—an ensemble of previous training states—looks at the virtual images and generates pseudo-labels. AMF adaptively determines whether to trust the bounding box coordinates or the classification labels provided by the teacher, allowing the "Student" model to learn from synthetic data without being misled by generative artifacts.

Experimental Evaluation

The VIL framework was evaluated on the HICO-Det and V-COCO benchmarks. To supplement the training set, the authors generated a virtual dataset ($D_v$) targeting rare categories, producing 40 images per rare interaction for HICO-Det.

Performance Gains

VIL demonstrated consistent improvements across various baseline architectures:

  • State-of-the-Art Results: When integrated with the GEN-VLKT model, VIL pushed performance to 34.21 mAP on HICO-Det.
  • Rare Category Improvement: The impact was most pronounced in rare subsets. For the QPIC model, VIL delivered a 1.49 mAP increase in rare interaction detection, representing a relative improvement of nearly 7%.
  • Training Efficiency: These gains were achieved with only 10 additional training epochs, highlighting the efficiency of the VIL strategy compared to training from scratch.

Discussion and Limitations

While VIL provides a powerful solution for the long-tail problem, it is not without challenges. The computational cost of generating and filtering thousands of images via Stable Diffusion is significant. Furthermore, while the AMF module mitigates label noise, generating complex scenes with multiple overlapping interactions remains a difficult task for current generative models. Finally, a subtle "domain gap" persists between synthetic and real-world images, though the MUSIC pipeline's Scene Similarity filter helps narrow this divide.

Conclusion

Virtual Image Learning (VIL) represents a significant step forward in addressing data scarcity in HOI detection. By combining sophisticated generative "guardrails" with adaptive pseudo-labeling, VIL allows models to learn from rare interactions that are otherwise difficult to capture in the real world. This model-agnostic approach provides a scalable path toward more robust and comprehensive visual understanding.

Summary updated

BibTeX
@inproceedings{fang2023improving,
  author = {Shuman Fang and Shuai Liu and Jie Li and Guannan Jiang and Xianming Lin and Rongrong Ji},
  title = {{Improving Human-Object Interaction Detection via Virtual Image Learning}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611735},
}
Figure