Translation-equivariant Representation in Recurrent Networks with a Continuous Manifold of Attractors

Wenhao Zhang, Ying Nian Wu, Si WuNeurIPS 2022 · OpenReview

Overview

This paper bridges Lie group theory and neuroscience to explain how recurrent neural circuits maintain faithful internal representations of a moving world. The authors demonstrate that Continuous Attractor Networks (CANs) naturally support equivariant representations and propose a biologically plausible mechanism—inspired by the fruit fly's brain—where specialized "speed neurons" implement abstract mathematical operators to update the network's state.

What Problem Does This Paper Address?

Equivariance is a fundamental property of perception: if a stimulus in the world transforms (e.g., an object moves or rotates), our internal neural representation should transform in a corresponding, predictable way. While "equivariant neural networks" are a hot topic in feedforward machine learning, how the brain's recurrent circuitry achieves this remains a mystery.

Specifically, the authors tackle two primary gaps:

  1. Recurrent Mechanism: How do recurrent dynamics self-consistently generate equivariant representations?
  2. Operator Representation: How does the brain represent abstract group operators (like "translate by $x$ amount")? To predict where a moving object will be next, the brain needs to represent not just the object's position, but the speed of the transformation itself.

Methods

The study focuses on a 1D translation Lie group acting on a stimulus (like heading direction). The core of their model is the Continuous Attractor Network (CAN), a canonical circuit model where excitatory neurons are arranged such that their collective activity forms a stable "bump" on a continuous manifold.

Key methodological steps include:

  • Lie Group Analysis: Using group theory to define the requirements for an equivariant neural population code.
  • Eigenanalysis: Performing stability analysis on CAN dynamics to show that the network naturally possesses a "neutrally stable" mode corresponding to translation along the stimulus manifold.
  • Translation Generators: Mathematically showing that the "generator" of a translation (the Lie algebra) can be mapped to the derivative of the recurrent weight matrix.
  • Biologically Plausible Implementation: Since synapses can't easily implement mathematical derivatives, the authors propose a multi-population model. They introduce "speed neurons" that provide shifted feedback to the CAN, effectively implementing the translation operator. This architecture is directly inspired by the Drosophila (fruit fly) central complex.
The concept of an equivariant map and how population responses shift under translation operators.

Key Insights and Ideas

  • Inherent Equivariance: A major insight is that CANs are naturally equivariant due to their translation-invariant recurrent connections. The network doesn't need to "learn" equivariance; it emerges directly from the circuit's symmetry.
  • Synaptic Encoding of Operators: The abstract mathematical concept of a "translation generator" is physically stored in the shift of feedback connections. In the fly brain, P-EN neurons provide this shifted input to E-PG neurons (the attractor network), allowing the "bump" of activity to move.
  • Disentangled Representations: The model separates the representation of the state (the stimulus position in the CAN) from the operator (the speed represented by separate neuronal populations). This disentanglement is crucial for tasks like path integration.
  • Path Integration as Group Action: The accumulation of movement (path integration) is shown to be equivalent to the composition of translation operators. Because the translation group is abelian (commutative), the order of movements doesn't change the final represented position.
Structure of a Continuous Attractor Network and its stability analysis, showing the manifold of attractors.

Key Results

  • Stability: Eigenanalysis confirms that the continuous manifold of attractors is stable against perturbations, ensuring the representation is robust to noise.
  • Speed Control: Simulations show that the difference in firing rates between "left-moving" and "right-moving" speed neurons linearly determines the translation speed of the neural representation, matching theoretical predictions.
  • Biological Match: The model's joint tuning curves (how neurons respond to both stimulus and speed) are consistent with experimental observations in the fruit fly's internal compass.
  • Commutative Integration: The network successfully performs path integration where the final state is independent of the order of speed inputs, satisfying the mathematical properties of the translation group.

Limitations and Future Work

  • Homogeneity Assumption: The model assumes perfectly homogeneous neurons and symmetric connections. Real biological circuits are noisy and heterogeneous, which might break "perfect" equivariance.
  • Dimensionality: The current work is limited to 1D translation. The authors suggest extending this to 2D (to explain grid cells in a torus manifold) and 3D rotations (non-commutative groups) as future directions.
  • Sensory Integration: The model focuses on internal "dead reckoning" (path integration). Future versions would need to incorporate external sensory cues to "pin" the representation to the real world and prevent drift.

Summary updated

Abstract

Equivariant representation is necessary for the brain and artificial perceptual systems to faithfully represent the stimulus under some (Lie) group transformations. However, it remains unknown how recurrent neural circuits in the brain represent the stimulus equivariantly, nor the neural representation of abstract group operators. The present study uses a one-dimensional (1D) translation group as an example to explore the general recurrent neural circuit mechanism of the equivariant stimulus representation. We found that a continuous attractor network (CAN), a canonical neural circuit model, self-consistently generates a continuous family of stationary population responses (attractors) that represents the stimulus equivariantly. Inspired by the Drosophila's compass circuit, we found that the 1D translation operators can be represented by extra speed neurons besides the CAN, where speed neurons' responses represent the moving speed (1D translation group parameter), and their feedback connections to the CAN represent the translation generator (Lie algebra). We demonstrated that the network responses are consistent with experimental data. Our model for the first time demonstrates how recurrent neural circuitry in the brain achieves equivariant stimulus representation.
BibTeX
@inproceedings{zhang2022translation,
  author = {Wenhao Zhang and Ying Nian Wu and Si Wu},
  title = {{Translation-equivariant Representation in Recurrent Networks with a Continuous Manifold of Attractors}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=F4AqCIOaYD},
}
Figure