NeSy4VRD: A Multifaceted Resource for Neurosymbolic AI Research using Knowledge Graphs in Visual Relationship Detection

David Herron, Ernesto Jiménez-Ruiz, Giacomo Tarroni, Tillman Weyde

TL;DR

NeSy4VRD bridges the gap between computer vision and the Semantic Web by providing a heavily cleaned, quality-improved version of the Visual Relationship Detection (VRD) dataset paired with a custom OWL ontology. Along with re-establishing public access to the original images, it introduces open-source Python infrastructure to easily load, extract, and extend these annotations within a knowledge graph, offering a benchmark for Neurosymbolic AI (NeSy) research.

What Problem Does This Paper Address?

Neurosymbolic AI (NeSy) aims to combine the data-driven learning capabilities of deep neural networks with the explicit, logical reasoning of symbolic AI. Web Ontology Language (OWL) ontologies and Knowledge Graphs (KGs) are ideal candidates for the symbolic side, but researchers face a major bottleneck: a severe lack of high-quality datasets that contain both image data for deep learning and well-aligned ontologies for logical reasoning.

To make matters worse, existing computer vision datasets are often too semantically messy for strict logical reasoning. The authors took a close look at the popular Visual Relationship Detection (VRD) dataset and found severe quality issues:

  • Stark label variability: The class bear referred to both real bears and teddy bears; plate meant dishware, license plates, and baseball home plates.
  • Redundant classes: Indistinguishable objects were split into separate classes like plane and airplane or street and road.
  • Ambiguous predicates: The relationship across was inconsistently used to mean "alongside of," "is crossing," or "is across from."

While deep learning models can sometimes tolerate this noise, strict deductive reasoning engines break down entirely. Furthermore, the original VRD images recently went offline, making the dataset inaccessible to new researchers.

Methods

The authors developed NeSy4VRD to systematically clean the VRD dataset and construct a robust companion ontology called VRD-World. Their approach consists of three main components:

  1. The VRD-World Ontology: A rich OWL ontology engineered to precisely mirror the dataset's domain. It includes hierarchical structures for both object classes and properties (relationships), enabling meaningful OWL reasoning.
  2. The NeSy4VRD Protocol: A custom, text-based declarative protocol that allows researchers to specify annotation changes (such as modifying bounding boxes, updating object classes, or adding/removing relationships) in simple text files.
  3. An Automated Workflow: A managed Python pipeline that processes these protocol files sequentially to update the dataset in an automated, repeatable manner.

Additionally, the toolkit includes sample Python code utilizing the RDFLib package to seamlessly load these visual relationship annotations into an OWL-hosted knowledge graph and extract them back into their native formats.

Datasets

The project builds directly upon the Visual Relationship Detection (VRD) dataset, which contains 4,000 training images and 1,000 test images.

The newly introduced NeSy4VRD resource includes:

  • Re-established public hosting of the original 5,000 VRD images (with permission from the original authors).
  • A completely overhauled set of annotations containing 109 object classes (up from 100) and 71 predicates (up from 70) to resolve semantic ambiguities.
Fig. 1. Two representative images from the VRD dataset with samples of representative annotated visual relationships as 3-tuples.

Key Insights and Ideas

  • Deep learning noise is symbolic poison: Computer vision datasets often prioritize scale over semantic precision. NeSy4VRD demonstrates that preparing data for symbolic AI requires a fundamentally different, much stricter approach to data cleaning.
  • Extensibility is a first-class requirement: The authors recognize that no ontology fits every research goal. By providing the protocol and workflow scripts, they allow future researchers to easily tailor the dataset and ontology to their specific logical constraints without starting from scratch.
  • Small datasets are a feature, not a bug: Because deep learning is data-hungry, the relatively small size of the VRD dataset (5,000 images) provides an ideal testing ground for zero-shot and few-shot learning, where symbolic knowledge graphs can help models generalize better from limited data.

Key Results

The annotation customization process applied corrections to 2,543 images—just over half of the entire dataset.

As shown in the paper's comparative metrics, the cleaning process successfully eliminated all duplicate visual relationships:

Table 1. Quantitative comparison of the NeSy4VRD and VRD visual relationship annotations. ‘Visual relationship’ has been abbreviated with the acronym VR.

The resulting VRD-World ontology provides a highly structured environment for reasoning, boasting 239 total classes and 815 axioms:

Extracted table from page 8 of NeSy4VRD: A Multifaceted Resource for Neurosymbolic AI Research using Knowledge Graphs in Visual Relationship Detection

Limitations and Future Work

As a resource paper, NeSy4VRD focuses entirely on dataset engineering and infrastructure rather than introducing a novel machine learning architecture. The authors do not provide benchmarks showing performance increases on visual relationship detection tasks within this text, though they note that their ongoing research utilizes this dataset to explore OWL and Datalog knowledge graph reasoning to enhance deep learning.

Future work includes leveraging this toolkit for symbolic knowledge injection and exploring how the rich ontology can improve zero-shot learning performance in computer vision models.

Summary updated

Abstract

NeSy4VRD is a multifaceted resource designed to support the development of neurosymbolic AI (NeSy) research. NeSy4VRD re-establishes public access to the images of the VRD dataset and couples them with an extensively revised, quality-improved version of the VRD visual relationship annotations. Crucially, NeSy4VRD provides a well-aligned, companion OWL ontology that describes the dataset domain.It comes with open source infrastructure that provides comprehensive support for extensibility of the annotations (which, in turn, facilitates extensibility of the ontology), and open source code for loading the annotations to/from a knowledge graph. We are contributing NeSy4VRD to the computer vision, NeSy and Semantic Web communities to help foster more NeSy research using OWL-based knowledge graphs.
BibTeX
@misc{herron2023nesy,
  author = {David Herron and Ernesto Jiménez-Ruiz and Giacomo Tarroni and Tillman Weyde},
  title = {{NeSy4VRD: A Multifaceted Resource for Neurosymbolic AI Research using Knowledge Graphs in Visual Relationship Detection}},
  year = {2023},
  eprint = {2305.13258},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13258},
}
Figure