Making Sense of Raw Input (Extended Abstract)

Richard Evans, Matko Bosnjak, Lars Buesing, Kevin Ellis, David Pfau, Pushmeet Kohli, Marek J. Sergot

Abstract: We present a neuro-symbolic framework that learns to interpret raw sensory data by jointly training a neural network for perception and a logic-based engine for reasoning. By encoding the neural network as a logic program, the system uses high-level symbolic constraints to resolve low-level perceptual ambiguities. Our results demonstrate 100% accuracy and perfect generalization in complex domains like Sokoban and MNIST sequence induction, achieving extreme data efficiency compared to standard deep learning approaches.

1. Introduction

The divide between sub-symbolic AI (neural networks) and symbolic AI (logic systems) remains a significant hurdle in the quest for Artificial General Intelligence. While neural networks excel at pattern recognition from raw data, they often function as "black boxes" and struggle with out-of-distribution generalization. Conversely, symbolic systems offer interpretability and formal reasoning but typically require human-engineered symbolic representations. This paper addresses the "apperception" problem: how an agent can autonomously construct an interpretable theory of its environment directly from noisy, unstructured sensory input.

2. The Neuro-Symbolic Apperception Engine

The proposed architecture integrates a Perceptual Front-end with a Symbolic Back-end. The front-end consists of a binary neural network that maps raw pixels into symbolic disjunctions—sets of possible classifications rather than single labels. The back-end is an unsupervised program synthesis system that induces first-order logic rules to explain the sequence of observed symbols.

The core innovation is the joint optimization of these components as a single satisfiability (SAT) problem. This allows the system to leverage "top-down" logical constraints to correct "bottom-up" perceptual errors. For instance, if a sequence of digits follows a mathematical progression, the logic engine can disambiguate a blurry image by selecting the only digit that fits the pattern.

3. Experimental Results

The framework was evaluated on three distinct tasks: modified Seek Whence sequence induction using MNIST digits, the puzzle game Sokoban, and ambiguous spatial data tasks.

Figure 1: Resolving visual ambiguity in MNIST sequences. The system identifies the correct digit by finding the interpretation that satisfies a consistent symbolic pattern.

3.1. Sequence Induction and Perception

In the MNIST-based Seek Whence tasks, the engine successfully resolved visual ambiguities. As shown in Figure 1, when presented with a digit that could be interpreted as multiple values (e.g., a '5' resembling a '6'), the system utilizes the induced logical theory of the sequence to determine the correct identity. This demonstrates how symbolic regularities can directly improve perceptual accuracy.

3.2. Efficiency and Generalization in Sokoban

In the Sokoban domain, the engine induced the complete set of game rules from fewer than 24 game traces. Unlike deep reinforcement learning models that require millions of frames, this system achieves 100% accuracy with minimal data. Furthermore, because the rules are expressed in first-order logic, the learned "physics" of the game generalizes perfectly to boards of any size, a feat rarely achieved by purely neural architectures.

4. Discussion and Conclusion

The primary strength of this approach is its inherent interpretability; the system produces human-readable rules that can be verified and audited. However, the computational cost of solving joint SAT problems remains a significant limitation, currently restricting the engine to discrete, relatively small environments. Future work will explore scaling these methods to continuous sensory streams, aiming for a robust synthesis of perception and high-level reasoning.

Summary updated

Abstract

How should a machine intelligence perform unsupervised structure discovery over streams of sensory input? One approach to this problem is to cast it as an apperception task. Here, the task is to construct an explicit interpretable theory that both explains the sensory sequence and also satisfies a set of unity conditions, designed to ensure that the constituents of the theory are connected in a relational structure. However, the original formulation of the apperception task had one fundamental limitation: it assumed the raw sensory input had already been parsed using a set of discrete categories, so that all the system had to do was receive this already-digested symbolic input, and make sense of it. But what if we don't have access to pre-parsed input? What if our sensory sequence is raw unprocessed information? The central contribution of this paper is a neuro-symbolic framework for distilling interpretable theories out of streams of raw, unprocessed sensory experience. First, we extend the definition of the apperception task to include ambiguous (but still symbolic) input: sequences of sets of disjunctions. Next, we use a neural network to map raw sensory input to disjunctive input. Our binary neural network is encoded as a logic program, so the weights of the network and the rules of the theory can be solved jointly as a single SAT problem. This way, we are able to jointly learn how to perceive (mapping raw sensory information to concepts) and apperceive (combining concepts into declarative rules).
BibTeX
@inproceedings{evans2022making,
  author = {Richard Evans and Matko Bosnjak and Lars Buesing and Kevin Ellis and David Pfau and Pushmeet Kohli and Marek J. Sergot},
  title = {{Making Sense of Raw Input (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/799},
}
Figure