Generative replay for compositional visual understanding in the prefrontal-hippocampal circuit

Philipp Schwartenbeck, Alon Baram, Yunzhe Liu, Shirley Mark, Timothy Müller, Raymond J. Dolan, Matthew Botvinick, Zeb Kurth‐Nelson, Timothy E.j. BehrensbioRxiv 2021 · DOI

TL;DR

This paper reveals that our brains construct visual understanding by forming compositional representations and using "replay" – rapid sequences of neural activity – to test hypotheses about scene configurations. This suggests a unified mechanism for how the hippocampus and prefrontal cortex handle diverse cognitive tasks like navigation, planning, and visual comprehension.

What Problem Does This Paper Address?

Understanding how we make sense of complex visual scenes is a fundamental challenge. While we know the hippocampus and prefrontal cortex are crucial for this, the specific neural computations involved remain elusive. Visual understanding seems different from tasks like spatial navigation or planning, yet researchers suspect deeper computational similarities. This paper bridges this gap by investigating how the brain builds representations of visual scenes and whether mechanisms like neural replay, previously linked to planning and memory, are also involved in constructing our understanding of the present visual world.

Methods

The researchers employed two key neuroimaging techniques:

  • fMRI (functional Magnetic Resonance Imaging): Used to investigate the neural representations of visual scenes. They analyzed brain activity patterns to see if they encoded compositional and relational information about scene elements.
  • MEG (Magnetoencephalography): Used to examine the temporal dynamics of these representations. Specifically, they looked for "replay" – rapid sequences of neural activity – to see if these sequences reflected hypothesis testing during visual construction.

They developed a novel visual construction task where participants had to mentally deconstruct and reconstruct visual silhouettes from basic building blocks. This task allowed them to probe for generalizable representations and the temporal dynamics of hypothesis generation.

Datasets

The study used data collected from human participants performing a custom-designed visual construction task. No external datasets were used for training or evaluation.

Key Insights and Ideas

  • Compositional Understanding: The brain doesn't just see objects; it understands their relationships. Representations in the prefrontal cortex and hippocampus encode how building blocks are combined (e.g., "on-top," "beside"), allowing for flexible understanding of novel scenes.
  • Replay for Hypothesis Testing: Neural replay, previously thought to be for planning futures or consolidating memories, is repurposed here to test hypotheses about the current visual scene. These replay sequences are not random; they follow an optimal order, starting with more predictable elements and gradually refining hypotheses.
  • Unified Neural Code: The findings suggest a shared neural mechanism across seemingly disparate functions like spatial navigation, model-based planning, and visual understanding, all potentially relying on a "cognitive map" of task structure.
  • "Silhouette Algebra": The study introduces a concept akin to "silhouette algebra," where neural representations can be manipulated algebraically to predict how building blocks combine in specific relational positions, demonstrating a compositional and conjunctive code.

Key Results

  • fMRI: Showed that representations in the medial prefrontal cortex (mPFC) and anterior hippocampus encoded compositional and relational information about building blocks, generalizing across different silhouettes. This "silhouette algebra" analysis predicted neural activity patterns.
  • MEG: Revealed rapid neural sequences, or replay, during the planning phase of the visual construction task. These sequences were not random but followed a structured pattern, starting with unspecific hypotheses and progressing to more refined ones, mirroring an optimal hypothesis-testing strategy. Replay sequences were observed to be predictive of correct scene construction.

Limitations and Future Work

  • Task Specificity: While the task is designed to probe general principles, it's a simplified representation of real-world visual understanding. Future work could explore these mechanisms in more complex and naturalistic visual environments.
  • Causality: While the study shows correlations between neural activity, replay, and task performance, direct causal links could be further investigated using techniques like TMS or lesion studies.
  • Generalizability of Replay: The study focused on replay in a constructive hypothesis-testing context. Further research could explore its role in other forms of visual inference and understanding.
  • Computational Models: The authors suggest that these findings can inform the development of more sophisticated computational models of visual understanding, potentially bridging the gap between neuroscience and artificial intelligence.

Summary updated

Abstract

Abstract Understanding the visual world is a constructive process. Whilst a frontal-hippocampal circuit is known to be essential for this task, little is known about the associated neuronal computations. Visual understanding appears superficially distinct from other known functions of this circuit, such as spatial reasoning and model-based planning, but recent models suggest deeper computational similarities. Here, using fMRI, we show that representations of a simple visual scene in these brain regions are relational and compositional – key computational properties theorised to support rapid construction of hippocampal maps. Using MEG, we show that rapid sequences of representations, akin to replay in spatial navigation and planning problems, are also engaged in visual construction. Whilst these sequences have previously been proposed as mechanisms to plan possible futures or learn from the past, here they are used to understand the present. Replay sequences form constructive hypotheses about possible scene configurations. These hypotheses play out in an optimal order for relational inference, progressing from predictable to uncertain scene elements, gradually constraining possible configurations, and converging on the correct scene configuration. Together, these results suggest a computational bridge between apparently distinct functions of hippocampal-prefrontal circuitry, and a role for generative replay in constructive inference and hypothesis testing.
BibTeX
@misc{schwartenbeck2021generative,
  author = {Philipp Schwartenbeck and Alon Baram and Yunzhe Liu and Shirley Mark and Timothy Müller and Raymond J. Dolan and Matthew Botvinick and Zeb Kurth‐Nelson and Timothy E.J. Behrens},
  title = {{Generative replay for compositional visual understanding in the prefrontal-hippocampal circuit}},
  year = {2021},
  doi = {10.1101/2021.06.06.447249},
  howpublished = {bioRxiv},
}
Figure