Abstract
Neural-Symbolic Integration (NSI) seeks to combine the perceptual strengths of deep learning with the logical reasoning of symbolic AI. However, the field lacks a standardized benchmark that provides both complex sensory data and formal logical structures. This paper proposes the Visual Genome repository as a foundational dataset for NSI research. By mapping image regions to structured scene graphs and WordNet IDs, Visual Genome enables the translation of visual data into first-order logic. We outline five challenge tasks—including visual entailment and scene graph estimation—to drive progress in hybrid AI systems that can both perceive and reason.
Introduction
Artificial Intelligence research is currently divided into two primary paradigms: sub-symbolic systems (deep learning), which excel at pattern recognition in raw data, and symbolic systems (classical AI), which excel at logical reasoning and rule-following. The integration of these two—Neural-Symbolic Integration—is considered a "holy grail" of the field.
A significant barrier to progress in NSI is the lack of a common, high-quality benchmark. Existing computer vision datasets like ImageNet or Microsoft COCO are optimized for object identification but lack the formal logical structure required to test reasoning. Conversely, traditional logic-based datasets often lack the complexity of real-world sensory input. This paper argues that the Visual Genome (VG) dataset provides the necessary bridge to fairly compare and advance neural-symbolic approaches.
The Visual Genome Dataset
The Visual Genome dataset consists of 108,249 images containing 4.2 million region descriptions and 1.8 million relationships (e.g., "man wearing hat"). Crucially, these annotations are mapped to WordNet IDs, allowing the data to be translated into First-Order Logic (FOL) knowledge bases.
As shown in the figure above, VG's "scene graphs" act as a formal representation of the visual world, providing a structured playground for AI systems to practice "lifting" sub-symbolic vector representations into symbolic logic.
Proposed Challenge Tasks
To push the boundaries of NSI, we propose five novel challenge tasks utilizing the Visual Genome repository:
- Visual Entailment: Determining if one image region logically implies or contradicts another (e.g., does a "car" entail a "tire"?).
- Scene Graph Estimation: Automatically generating a complete graph of objects, attributes, and relationships from raw pixels.
- Visual Rule Extraction and Analogy: Mining data for logical rules (e.g., "If a man is not standing, he is likely sitting") and solving visual analogies.
- Collective Classification: Predicting labels for multiple objects simultaneously by leveraging their inter-relationships.
- Unsupervised Co-training: Using generative models to reconstruct images from symbols and vice versa.
Neural-Symbolic Integration Insights
The core challenge of NSI is the "lifting" problem—transforming raw sensory input into symbolic representations. Moving from simple perception (detecting a dog) to cognition (understanding that a dog is chasing a ball) requires operations that are inherently neural-symbolic. Unlike Statistical Relational Learning (SRL) or Inductive Logic Programming (ILP), which often struggle with either raw sensory input or noisy real-valued vectors, these proposed tasks require a hybrid approach capable of handling both.
Limitations and Future Work
While Visual Genome offers a robust framework, it is not without challenges. The "variable binding problem"—handling multiple instances of the same object type in different contexts—remains a significant technical hurdle. Furthermore, generating a full scene graph from scratch is an incredibly difficult task that will require better integration of attention-memory mechanisms and recurrent architectures.
Conclusion
By adopting Visual Genome as a common challenge, the NSI community can move toward a standardized evaluation of models. This transition from purely perceptual tasks to cognitive reasoning is essential for the development of AI that can truly see, understand, and think about the world.
Summary updated