Care about you: towards large-scale human-centric visual relationship detection

Bohan Zhuang, Qi Wu, Chunhua Shen, Ian D. Reid, Anton van den HengelarXiv 2017 · OpenReview

TL;DR

This paper introduces HCVRD, a large-scale dataset for human-centric visual relationship detection, featuring a vast number of relationship categories that naturally lead to a long-tail distribution. To address this challenge and enable zero-shot learning, they propose a novel webly-supervised approach that leverages web-crawled data to learn robust relationship representations.

What Problem Does This Paper Address?

Visual relationship detection aims to identify how objects in an image interact. A particularly interesting and practically relevant subset of this is human-object interactions (HOI). Understanding these interactions is crucial for applications like human behavior analysis and affordance recognition. However, existing datasets often simplify the problem by focusing on a limited number of interactions or objects. The reality of human-object interactions is far richer, with humans exhibiting a wide range of actions and relationships with objects. This complexity leads to a "long-tail" distribution in the data, where a few relationships are very common, but the vast majority are rare or even unseen during training. This poses a significant challenge for standard deep learning models that rely on abundant training data for each category. Furthermore, the ability to detect relationships not seen during training (zero-shot learning) is essential for real-world applicability.

Methods

The authors propose a webly-supervised relationship detection (WSRD) model. This model consists of two main components:

  1. Object Detection Module: This module, based on Faster R-CNN, is responsible for detecting both human subjects and objects within an image, along with their bounding boxes.
  2. Distance Metric Learning Module: This is the core of their novel approach. It takes the detected human-object pairs and compares them to a large collection of web-crawled visual relationship data. The goal is to learn a metric space where similar relationships are close together. This module involves:
  3. Web Data Crawling: Automatically collecting images from Google Images for each relationship category.
  4. Weakly-Supervised Filtering: A noise-robust approach to filter out irrelevant or mislabeled images from the web data, using random grouping and an attention mechanism.
  5. Deep Metric Learning: Learning an embedding space where positive pairs (human-object from HCVRD and a matching web image) are close, and negative pairs are far apart. This is achieved using a structured loss function on sampled positive and negative pairs.

The two modules are trained end-to-end, sharing convolutional layers for efficiency.

Datasets

  • HCVRD (Human-Centric Visual Relationship Detection) Dataset: This is a newly constructed, large-scale dataset introduced in the paper. It is built upon the Visual Genome dataset but specifically focuses on human-object interactions.
  • Key Features: Nearly 10,000 relationship categories (927 predicates and 1,824 object categories), including fine-grained human sub-categories (man, woman, boy, girl). It explicitly includes 18,471 "zero-shot" relationships that do not appear in the training set.
  • Comparison: Significantly larger in terms of relationship categories compared to previous datasets like HICO [2] and Verbs-COCO [8]. It also provides bounding boxes for both humans and objects.

Key Insights and Ideas

  • The Long-Tail is Reality: The paper strongly emphasizes that real-world human-object interactions exhibit a severe long-tail distribution, making it impractical to rely solely on fully supervised methods trained on limited data.
  • Web Data as a Solution: Leveraging vast amounts of noisy web data, combined with robust filtering and metric learning, is a promising direction for tackling the long-tail and zero-shot problems in visual relationship detection.
  • Zero-Shot is Crucial: The inclusion of a significant number of zero-shot relationships in HCVRD highlights the necessity of models that can generalize to unseen interactions.
  • Human Agency Matters: Differentiating human sub-categories (man, woman, boy, girl) is important because gender and age can influence interaction patterns.

Key Results

  • The proposed webly-supervised model (Ours) significantly outperforms baseline methods like Multilabel classification and JointCNN on the HCVRD dataset, particularly in predicate and relationship detection tasks.
  • On the non-zero-shot test set, the proposed model achieves Recall@100 of 31.08% for top-1 predicate detection and 48.98% for top-3, and 10.03% for top-1 relationship detection and 13.05% for top-3.
  • For the zero-shot benchmark, the model achieves Recall@100 of 8.57% for top-1 predicate detection and 13.42% for top-3, and 0.92% for top-1 relationship detection and 1.84% for top-3, outperforming baselines.
  • Ablation studies show that both the metric learning module and the use of web data are critical for achieving strong performance, with the metric learning module having a more substantial impact.

Limitations and Future Work

  • Computational Complexity: The paper acknowledges that the proposed method has high computational complexity, especially in the zero-shot setting due to the extensive web data search.
  • Web Data Quality: While they employ filtering, the inherent noise in web-crawled data remains a challenge. The paper notes that noise filtering has a less significant impact than metric learning, suggesting that common relationships in web searches are relatively clean.
  • Future Work: The authors suggest that their approach provides a strong baseline for future work on large-scale human-centric visual relationship detection, particularly for tackling long-tail and zero-shot scenarios. The dataset itself is a valuable resource for further research in this area.

Summary updated

Abstract

Visual relationship detection aims to capture interactions between pairs of objects in images. Relationships between objects and humans represent a particularly important subset of this problem, with implications for challenges such as understanding human behaviour, and identifying affordances, amongst others. In addressing this problem we first construct a large-scale human-centric visual relationship detection dataset (HCVRD), which provides many more types of relationship annotation (nearly 10K categories) than the previous released datasets. This large label space better reflects the reality of human-object interactions, but gives rise to a long-tail distribution problem, which in turn demands a zero-shot approach to labels appearing only in the test set. This is the first time this issue has been addressed. We propose a webly-supervised approach to these problems and demonstrate that the proposed model provides a strong baseline on our HCVRD dataset.
BibTeX
@misc{zhuang2017care,
  author = {Bohan Zhuang and Qi Wu and Chunhua Shen and Ian D. Reid and Anton van den Hengel},
  title = {{Care about you: towards large-scale human-centric visual relationship detection}},
  year = {2017},
  eprint = {1705.09892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.09892},
}
Figure