Constructing a Visual Relationship Authenticity Dataset

Chenhui Chu, Yuto Takebayashi, Mishra Vipul, Yuta NakashimaarXiv 2020 · arXiv

Introduction

Visual Relationship Detection (VRD) is the task of identifying how objects in an image interact, usually expressed as a triplet: (subject; predicate; object). While large-scale datasets like Visual Genome exist, they primarily contain "true" relationships. This lack of negative examples often leads to AI "hallucinations," where models predict relationships based on statistical commonality (e.g., "bike on road") rather than actual visual evidence.

Methodology

Researchers from Osaka University developed a multi-stage pipeline to construct the Visual Relationship Authenticity Dataset, explicitly labeling both true and false relationships.

The multi-stage pipeline for dataset construction, utilizing automated dependency parsing and human verification via AMT.
  1. Candidate Generation: Using the Flickr30k entities dataset, the team generated pairs of entities mentioned in captions. The text between these entities served as candidate predicates.
  2. Automated Filtering: A dependency parser analyzed caption structures to identify valid relationships. While efficient, this method only captured about 50% of true relationships.
  3. Crowdsourcing (AMT): Remaining candidates were sent to Amazon Mechanical Turk. Human workers labeled relationships as "True" or "False" based on the image contents.
Example of the source data from Flickr30k, where entities in captions are grounded to specific image regions.

Key Results and Dataset Scale

The resulting benchmark contains 31,769 images and 266,423 relationship instances. Unlike previous datasets, it includes an "Authenticity" column, marking nearly 70,000 instances as False.

Quality control was rigorous; AMT workers achieved a high F-score of 94.9%, ensuring the reliability of the crowdsourced labels. This "negative space" of visual data allows models to learn to reject incorrect but plausible-sounding descriptions.

Conclusion

By providing a massive resource of both true and false visual relationships, this dataset enables the training of more robust models that are less prone to hallucination. Future work will focus on improving coreference handling and linguistic parsing to further refine the detection of visually grounded truths.

Summary updated

Abstract

A visual relationship denotes a relationship between two objects in an image, which can be represented as a triplet of (subject; predicate; object). Visual relationship detection is crucial for scene understanding in images. Existing visual relationship detection datasets only contain true relationships that correctly describe the content in an image. However, distinguishing false visual relationships from true ones is also crucial for image understanding and grounded natural language processing. In this paper, we construct a visual relationship authenticity dataset, where both true and false relationships among all objects appeared in the captions in the Flickr30k entities image caption dataset are annotated. The dataset is available at https://github.com/codecreator2053/VR_ClassifiedDataset. We hope that this dataset can promote the study on both vision and language understanding.
BibTeX
@misc{chu2020constructing,
  author = {Chenhui Chu and Yuto Takebayashi and Mishra Vipul and Yuta Nakashima},
  title = {{Constructing a Visual Relationship Authenticity Dataset}},
  year = {2020},
  eprint = {2010.05185},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.05185},
}
Figure