Introduction
Visual Relationship Detection (VRD) is the task of identifying how objects in an image interact, usually expressed as a triplet: (subject; predicate; object). While large-scale datasets like Visual Genome exist, they primarily contain "true" relationships. This lack of negative examples often leads to AI "hallucinations," where models predict relationships based on statistical commonality (e.g., "bike on road") rather than actual visual evidence.
Methodology
Researchers from Osaka University developed a multi-stage pipeline to construct the Visual Relationship Authenticity Dataset, explicitly labeling both true and false relationships.
- Candidate Generation: Using the Flickr30k entities dataset, the team generated pairs of entities mentioned in captions. The text between these entities served as candidate predicates.
- Automated Filtering: A dependency parser analyzed caption structures to identify valid relationships. While efficient, this method only captured about 50% of true relationships.
- Crowdsourcing (AMT): Remaining candidates were sent to Amazon Mechanical Turk. Human workers labeled relationships as "True" or "False" based on the image contents.
Key Results and Dataset Scale
The resulting benchmark contains 31,769 images and 266,423 relationship instances. Unlike previous datasets, it includes an "Authenticity" column, marking nearly 70,000 instances as False.
Quality control was rigorous; AMT workers achieved a high F-score of 94.9%, ensuring the reliability of the crowdsourced labels. This "negative space" of visual data allows models to learn to reject incorrect but plausible-sounding descriptions.
Conclusion
By providing a massive resource of both true and false visual relationships, this dataset enables the training of more robust models that are less prone to hallucination. Future work will focus on improving coreference handling and linguistic parsing to further refine the detection of visually grounded truths.
Summary updated