Abstract
Neurosymbolic AI (NeSy) is currently a fragmented field where researchers often build custom datasets to showcase their specific models, making fair comparisons nearly impossible. This paper proposes a standardized "thought process" for benchmarking NeSy systems, using multi-label classification with background knowledge as a blueprint for how to define tasks, select datasets, and measure success through both performance and architectural efficiency.
Introduction
Neurosymbolic AI (NeSy) aims to bridge the gap between the perceptual capabilities of neural networks and the logical reasoning of symbolic systems. While the field has seen a surge in novel architectures, it suffers from a "fragmented evaluation landscape." Because there is no unified taxonomy or standard benchmarking suite, researchers frequently develop "bespoke" benchmarks that inadvertently favor their specific architectural strengths. This lack of standardization prevents the community from answering two fundamental questions: how well does a system perform a task (the Outside View), and how much does the integration of symbolic components actually contribute to that performance (the Inside View)?
The Problem: Bespoke Benchmarking
In the current research environment, a system designed for heavy logical reasoning might be tested on a dataset featuring simple images but complex rules. Conversely, a vision-centric system might be evaluated on complex images with trivial logical constraints. This inconsistency makes it impossible to compare models head-to-head. To move forward, the field requires a principled approach to benchmarking that accounts for both sub-symbolic feature complexity and symbolic structural complexity.
A Formal Framework for NeSy Evaluation
The authors propose a three-pillar framework to standardize the benchmarking process:
1. Task Formalism
They provide a rigorous mathematical definition for "Multi-label classification with background knowledge." By defining tasks using propositional logic formulas ($\alpha$), they create a language-agnostic method to describe constraints that any valid output must satisfy. This allows for a clear distinction between the raw data and the logical rules governing the domain.
2. Dual-View Evaluation
Benchmarks must move beyond simple accuracy. The authors argue for a dual-view approach:
- The Outside View: Measuring observable behavior and task accuracy.
- The Inside View: Analyzing how the agent is designed and how the neural and symbolic components interact.
3. Metric Pairing
To satisfy the dual-view, performance metrics must be paired with control metrics. For instance, pairing Collective Accuracy with Network Size (number of trainable parameters) prevents "cheating," where a massive neural network simply memorizes constraints that the symbolic component was intended to handle.
The Landscape of NeSy Datasets
The paper categorizes existing NeSy datasets into three distinct patterns:
- Symbolic: Reasoning tasks converted into learning tasks, such as finding the shortest path in a graph.
- Compositional: Tuples of sub-symbolic data constrained by a specific structure, such as MNIST Sudoku.
- Hierarchical: Classes organized in a logical hierarchy, such as ImageNet classes mapped to WordNet.
A significant finding is the inverse relationship between feature complexity and logical complexity in current datasets. Most benchmarks occupy the extremes: complex images with simple logic or simple images with complex logic. The authors identify a "Complexity Void" where both sub-symbolic features (e.g., raw video) and symbolic structures (e.g., complex traffic laws) are highly complex. They cite the Road-R dataset—autonomous driving videos with logical requirements—as a rare example that addresses this void.
Key Metrics for Success
Beyond standard accuracy, the authors emphasize Semantic Consistency. This metric evaluates whether a system's output makes sense according to the predefined logical rules, regardless of whether the answer is "correct" in the ground-truth sense. This is vital for understanding if the symbolic integration is functioning as intended.
Ultimately, the authors call for a benchmarking suite similar to GLUE in the Natural Language Processing world. Such a suite would require NeSy systems to demonstrate a diverse range of capabilities—logic, perception, and generalization—rather than excelling in a single, isolated dimension.
Conclusion and Future Work
As a position paper, this work does not seek to establish new state-of-the-art accuracy records. Instead, it provides the conceptual "meta-rules" for how the NeSy community should build and evaluate future systems. While the current focus is on multi-label classification, the authors believe this thought process is transferable to other domains like reinforcement learning and generative modeling. Future work will involve validating this approach across a wider variety of tasks and developing standards for representing and sharing NeSy-specific background knowledge.
Summary updated