Composing Neural Learning and Symbolic Reasoning with an Application to Visual Discrimination

Adithya Murali, Atharva Sehgal, Paul Krogmeier, P. Madhusudan

TL;DR

This paper introduces Visual Discrimination Puzzles (VDPs), a new task requiring interpretable discriminators for visual concepts. They propose a neuro-symbolic framework that combines neural object detection with symbolic logic to generate these discriminators, achieving strong performance on complex visual reasoning tasks.

What Problem Does This Paper Address?

The paper tackles the challenge of integrating pre-trained neural models to solve high-level cognitive tasks that have clear specifications, especially in low-data regimes. While deep learning excels at specialized tasks, composing these models for complex reasoning with verifiable outputs remains difficult. Specifically, they introduce Visual Discrimination Puzzles (VDPs) where the goal is to find a property that distinguishes a set of example images from a set of candidate images, and this property must be interpretable and verifiable. This is distinct from tasks like Visual Question Answering (VQA) which typically query a single image.

Methods

The authors propose a two-phase neuro-symbolic framework:

  • Phase 1: Neural Learning: A neural network (specifically YOLOv4) is trained offline on large datasets to detect objects, their attributes, and relationships within images, generating "scene graphs."
  • Phase 2: Symbolic Reasoning: A novel interface, "First-Order Scene Logic" (FO-SL), is used to represent these scene graphs. A symbolic learner then synthesizes a discriminator expressed in FO-SL. This logic is designed to be robust to variations in neural model outputs and ensures that discriminators remain valid even if new objects or relationships are detected. The synthesis process uses a SAT solver to efficiently search for these logical discriminators.

Datasets

The paper introduces and uses several datasets:

  • Natural Scenes VDP dataset: ~3864 puzzles created from 20 base real-world concept classes.
  • GQA VDP dataset: ~5000 puzzles automatically generated from the GQA dataset, leveraging its scene graphs and questions.
  • CLEVR VDP dataset: 825 puzzles generated from the CLEVR domain, featuring abstract VDPs with unique minimal discriminators in FO-SL.
  • ODDONE Puzzles dataset: 1872 puzzles for an "odd one out" task, adapted from CLEVR.

Key Insights and Ideas

  • VDPs as a bridge: VDPs offer a compelling task that requires both visual perception and symbolic reasoning, bridging the gap between perception and higher-level cognition.
  • Interpretability is key: The demand for interpretable discriminators is crucial for understanding why a model makes a certain decision, moving beyond black-box predictions.
  • FO-SL's extension property: The proposed First-Order Scene Logic (FO-SL) is designed to be robust to the inherent variability and incompleteness of neural vision models, ensuring that learned discriminators remain valid as the vision system improves.
  • Neuro-symbolic composition: The framework demonstrates that combining neural perception with symbolic logic synthesis can effectively solve complex reasoning tasks, especially in few-shot scenarios.
  • Neural baselines struggle: Purely neural approaches (image similarity, prototypical networks) perform poorly on VDPs, highlighting the need for explicit symbolic reasoning to capture the logical specifications.

Key Results

  • The proposed neuro-symbolic framework achieves significant performance on the VDP tasks:
  • 68% accuracy on the Natural Scenes dataset, with an average of 71% per concept class.
  • 81% accuracy on the GQA VDP dataset when using ground-truth scene graphs.
  • 99% accuracy on the CLEVR VDP dataset, where discriminators are unique by construction.
  • The framework generates interpretable discriminators, often with multiple quantifiers and complex nesting.
  • Ablation studies show that FO-SL is crucial for robustness and naturalness, outperforming full first-order logic which can lead to unnatural discriminators.
  • Neural baselines achieved only around 40% accuracy on the CLEVR VDP dataset, significantly underperforming the neuro-symbolic approach.

Limitations and Future Work

  • Vision model failures: The primary failure mode of the system is the inaccuracy or incompleteness of the underlying neural vision model.
  • Expressive power of interface: The current scene graph representation has limitations in expressing certain spatial relationships (e.g., "bottom portion of the image").
  • Expressive power of FO-SL: While robust, FO-SL's conjunctive fragment cannot express all desired discriminators (e.g., those requiring negation or unguarded quantification).
  • Future Work:
    • Developing frameworks more robust to vision model failures.
  • Exploring more expressive logics or interfaces.
  • Adapting the approach to generate natural language discriminators.
  • Applying the neuro-symbolic approach to real-world applications like medical diagnosis and robotics.

Summary updated

Abstract

We consider the problem of combining machine learning models to perform higher-level cognitive tasks with clear specifications. We propose the novel problem of Visual Discrimination Puzzles (VDP) that requires finding interpretable discriminators that classify images according to a logical specification. Humans can solve these puzzles with ease and they give robust, verifiable, and interpretable discriminators as answers. We propose a compositional neurosymbolic framework that combines a neural network to detect objects and relationships with a symbolic learner that finds interpretable discriminators. We create large classes of VDP datasets involving natural and artificial images and show that our neurosymbolic framework performs favorably compared to several purely neural approaches.
BibTeX
@inproceedings{murali2022composing,
  author = {Adithya Murali and Atharva Sehgal and Paul Krogmeier and P. Madhusudan},
  title = {{Composing Neural Learning and Symbolic Reasoning with an Application to Visual Discrimination}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=4qI8PlrEbTY},
}
Figure