Toward a Unified Sub-symbolic Computational Theory of Cognition

Martin V. ButzFrontiers in Psychology 2016 · DOI

Abstract

This paper proposes a unifying, sub-symbolic, computational theory of cognition that integrates psychological theories, artificial intelligence, and neuroscience. By leveraging the free energy principle, the framework explains how abstract cognitive knowledge is learned from actively gathered sensorimotor experiences. It posits that symbols are emergent neural attractors grounded in physical interactions, offering a quantitative bridge between low-level processing and high-level thought.

Introduction: Bridging the Cognitive Divide

For decades, cognitive science has grappled with a fundamental split. Embodied Cognition (EC) argues that thinking is deeply rooted in sensory experiences and physical bodies, while Classical AI relies on symbolic logic and production rules. The "missing link" is a formal computational bridge: how does a brain transition from raw sensorimotor data—like the weight of an object—to abstract concepts like "containment"?

Martin Butz addresses this by proposing a sub-symbolic theory where "symbols" are stable patterns of neural activity within a predictive, generative model. This framework synthesizes theories of event segmentation, anticipatory behavior, and predictive coding into a single, mathematically grounded architecture.

The Predictive Processing Loop

At the heart of this theory is a continuous loop of predictive processing. Borrowing from Karl Friston’s free energy principle, the brain is viewed as a generative model striving to minimize prediction error (free energy).

The predictive processing loop integrating prior and posterior encodings.

The loop functions through three fundamental types of predictive encodings:

  1. Spatial: Mapping frames of reference (e.g., hand position relative to gaze).
  2. Top-down: Generalizing features and "Gestalt" perceptions (e.g., object recognition).
  3. Temporal: Predicting how actions and forces cause changes over time.

Neural activity-oriented inference causes the continuous adaptation of these active encodings, where top-down predictions and bottom-up error signals interact to settle into a "global posterior" state. Meanwhile, neural structure-oriented inference leads to long-term adaptation of the generative model as a whole.

Grounded Semantics and Symbols as Attractors

A key insight of this framework is that "symbols" and "thoughts" are not hard-coded primitives but distributed neural attractors. When a system perceives or imagines a situation, such as a "ball in a bowl," it activates a set of predictive encodings that reach a local free-energy minimum through mutual consistency.

Compositional concept of a 'ball in a bowl' as a distributed neural attractor.

Abstract concepts are thus grounded in layers of simpler sensorimotor episodes. For instance, the concept of "in" (containment) is learned through physical experiences of objects being placed in containers. Imagination and abstract thought occur when the system "decouples" from actual sensory input, allowing the internal generative model to run freely to explore hypothetical scenarios or future states.

Core Propositions of the Theory

The theory is summarized by nine main propositions that define its scope:

  1. Modularity: The brain is a modular, probabilistic, predictive system.
  2. Encoding Types: Encodings are categorized as temporal, spatial, and top-down.
  3. Abstraction: Modularity enables effective hierarchical abstraction.
  4. Anticipation: Thought is inherently anticipatory and driven by homeostasis.
  5. Concepts as Minima: Concepts are consistent free-energy minima (attractors).
  6. Compositionality: Relating these minima allows for complex, compositional thought.
  7. Scenarios: Perceived situations are compositional concepts.
  8. Episodes: Continuous experience is segmented into concatenations of event schemata.
  9. Ideas: Thinking is the active exploration of these conceptual spaces.

Future Directions and Limitations

While providing a robust roadmap, the theory faces several challenges. A "fully integrative implementation" combining all proposed biases does not yet exist. Furthermore, training such an agent requires massive amounts of embodied data. The author suggests using high-fidelity physics simulators (like MuJoCo) or game engines as "digital wombs" for cognitive development.

Future work must also address the social and linguistic dimensions of cognition—specifically how social interaction shapes these neural attractors and how the system might encode the intentions and knowledge of other agents in the environment.

Summary updated

Abstract

This paper proposes how various disciplinary theories of cognition may be combined into a unifying, sub-symbolic, computational theory of cognition. The following theories are considered for integration: psychological theories, including the theory of event coding, event segmentation theory, the theory of anticipatory behavioral control, and concept development; artificial intelligence and machine learning theories, including reinforcement learning and generative artificial neural networks; and theories from theoretical and computational neuroscience, including predictive coding and free energy-based inference. In the light of such a potential unification, it is discussed how abstract cognitive, conceptualized knowledge and understanding may be learned from actively gathered sensorimotor experiences. The unification rests on the free energy-based inference principle, which essentially implies that the brain builds a predictive, generative model of its environment. Neural activity-oriented inference causes the continuous adaptation of the currently active predictive encodings. Neural structure-oriented inference causes the longer term adaptation of the developing generative model as a whole. Finally, active inference strives for maintaining internal homeostasis, causing goal-directed motor behavior. To learn abstract, hierarchical encodings, however, it is proposed that free energy-based inference needs to be enhanced with structural priors, which bias cognitive development toward the formation of particular, behaviorally suitable encoding structures. As a result, it is hypothesized how abstract concepts can develop from, and thus how they are structured by and grounded in, sensorimotor experiences. Moreover, it is sketched-out how symbol-like thought can be generated by a temporarily active set of predictive encodings, which constitute a distributed neural attractor in the form of an interactive free-energy minimum. The activated, interactive network attractor essentially characterizes the semantics of a concept or a concept composition, such as an actual or imagined situation in our environment. Temporal successions of attractors then encode unfolding semantics, which may be generated by a behavioral or mental interaction with an actual or imagined situation in our environment. Implications, further predictions, possible verification, and falsifications, as well as potential enhancements into a fully spelled-out unified theory of cognition are discussed at the end of the paper.
BibTeX
@inproceedings{butz2016toward,
  author = {Martin V. Butz},
  title = {{Toward a Unified Sub-symbolic Computational Theory of Cognition}},
  booktitle = {Frontiers in Psychology},
  year = {2016},
  doi = {10.3389/fpsyg.2016.00925},
}
Figure