HCVRD: A Benchmark for Large-Scale Human-Centered Visual Relationship Detection

Bohan Zhuang, Qi Wu, Chunhua Shen, Ian D. Reid, Anton van den Hengel

Introduction

Visual Relationship Detection (VRD) is a fundamental task in computer vision that involves identifying triplets of [subject, predicate, object] within an image, such as "man riding bicycle" or "woman holding umbrella." While general VRD has seen significant progress, human-centered interactions present unique challenges. Humans possess agency and interact with their environment in a vast, nuanced variety of ways that are often difficult for standard models to capture.

A primary hurdle in this domain is the "long-tail" distribution of visual relationships. In real-world scenarios, a few common interactions (e.g., "wearing shirt") appear frequently, while thousands of others (e.g., "girl playing with kite") are extremely rare. Most existing datasets are relatively small or "sanitized," focusing on a limited set of common actions. This paper introduces HCVRD, a massive new benchmark designed to reflect the true complexity of human interactions, and proposes a webly-supervised metric learning framework to address the resulting zero-shot detection challenges.

Figure 1: The long-tail distribution of relationship categories in the HCVRD dataset, illustrating the variety of human-centric interactions.

The HCVRD Dataset

The Human-Centered Visual Relationship Detection (HCVRD) dataset is the primary contribution of this work. Derived and refined from Visual Genome, it contains 52,855 images featuring 1,824 object categories and 927 predicates. This results in a total of 9,852 unique relationship types.

Crucially, the dataset is designed to test a model's ability to generalize. It includes a zero-shot split consisting of 18,471 relationships that never appear in the training data. This setup forces models to move beyond simple pattern matching and toward a more semantic understanding of interactions. Furthermore, the authors found that categorizing "human" into more fine-grained types—man, woman, boy, and girl—provides strong priors that improve detection accuracy, as certain interactions are more statistically likely for specific age and gender groups.

Webly-supervised Relationship Detection (WSRD)

To tackle the nearly 10,000 relationship types in HCVRD, the authors propose the Webly-supervised Relationship Detection (WSRD) model. Traditional "softmax" classifiers struggle with such a large and dynamic label space, especially for zero-shot categories. Instead, WSRD utilizes a metric learning approach.

Architecture Overview

The model consists of three core components:

  1. Feature Extraction & Detection: A Faster R-CNN style architecture is employed to detect humans and objects within the scene.
  2. Webly-Supervised Metric Learning: The model learns a semantic feature embedding where similar interactions are mapped close together in a high-dimensional space.
  3. Noise-Robust Web Crawling: To populate this embedding space, the authors crawled approximately 788,000 images from the web. To mitigate the "noise" inherent in web data (e.g., irrelevant or mislabeled images), an attentional pooling mechanism filters out low-confidence samples.
Figure 5: The WSRD architecture, featuring an object detection module and a webly-supervised metric learning module that leverages web data for feature embedding.

Inference via Nearest Neighbor

During inference, the model does not simply assign a class label. Instead, it extracts a feature vector from the input image and performs a nearest-neighbor search within the massive database of cleaned web-crawled images. By identifying the most similar web image (e.g., a "girl playing with kite"), the model can predict relationships it has never encountered in its formal training set. This non-parametric approach allows the system to scale to a nearly infinite number of categories without requiring architecture retraining.

Results and Discussion

The WSRD model demonstrates significant improvements over traditional baselines. On the full HCVRD dataset, it achieved a Recall@100 (top-3) of 48.98% for predicate detection, nearly doubling the performance of standard "JointCNN" models which falter under the weight of the large label space.

In the challenging zero-shot setting, the model achieved a Recall@100 of 13.42%. While this number may seem low in isolation, it represents a major milestone; traditional classifiers typically score 0% on unseen classes. Ablation studies further confirmed that the noise-filtering step for web data is vital, as accuracy dropped noticeably when "messy" data was included in the reference set.

Conclusion

HCVRD provides a rigorous benchmark for the next generation of human-centric visual understanding. By embracing the long-tail distribution of real-world interactions and leveraging web-scale data through metric learning, the WSRD framework offers a viable path toward robust zero-shot relationship detection. Future work may address semantic ambiguities (synonyms) in evaluation and the computational overhead of large-scale nearest-neighbor searches, potentially extending these concepts to temporal data in video sequences.

Summary updated

Abstract

Visual relationship detection aims to capture interactions between pairs of objects in images. Relationships between objects and humans represent a particularly important subset of this problem, with implications for challenges such as understanding human behavior, and identifying affordances, amongst others. In addressing this problem we first construct a large-scale human-centric visual relationship detection dataset (HCVRD), which provides many more types of relationship annotations (nearly 10K categories) than the previous released datasets. This large label space better reflects the reality of human-object interactions, but gives rise to a long-tail distribution problem, which in turn demands a zero-shot approach to labels appearing only in the test set. This is the first time this issue has been addressed. We propose a webly-supervised approach to these problems and demonstrate that the proposed model provides a strong baseline on our HCVRD dataset.
BibTeX
@inproceedings{zhuang2018hcvrd,
  author = {Bohan Zhuang and Qi Wu and Chunhua Shen and Ian D. Reid and Anton van den Hengel},
  title = {{HCVRD: A Benchmark for Large-Scale Human-Centered Visual Relationship Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12260},
}
Figure