Implementing Inductive bias for different navigation tasks through diverse RNN attrractors

Tie Xu, Omri Barak

TL;DR

Researchers have demonstrated that the "inductive bias" needed for different navigation tasks—like following a map versus remembering landmarks—can be hardcoded into a Recurrent Neural Network (RNN) through its attractor landscapes. By using a two-stage training process, they showed that "PosNets" (which form continuous plane attractors) excel at metric spatial tasks, while "MemNets" (which form discrete fixed-point attractors) are better at topological tasks. Combining them into a modular system yields the best of both worlds.

What Problem Does This Paper Address?

In both biology and AI, navigation is often assumed to require a "cognitive map"—usually a metric, Euclidean representation of space. However, different environments demand different strategies. If you’re navigating a city grid, you need precise coordinates (metric regularity). If you’re following a trail of breadcrumbs or landmarks, you need to remember the sequence of "what" you saw last (topological regularity).

The challenge is that standard end-to-end reinforcement learning often struggles to find the "right" internal representation for these diverse tasks. This paper asks: How can we implement specific inductive biases in neural networks to handle different types of navigation, and what do those internal representations actually look like?

Methods

The authors propose a two-stage learning framework inspired by biological evolution and development:

  1. Task-Agnostic Pre-training: Instead of training for a specific goal, they first shape the RNN's internal connectivity. They use an objective function to force the network to represent either its current position (PosNet) or the identity of the last landmark it saw (MemNet). A third group, RanNet, is left with random internal weights as a control.
  2. Task-Specific Q-Learning: After pre-training, the internal "brain" of the network is frozen. Only the "readout" weights (the part that translates internal thoughts into actions like "move North") are trained using Reinforcement Learning (specifically, a fast recursive least squares version of Q-learning).

Finally, they introduce a Modular System that concatenates a PosNet and a MemNet, using a "selector" layer to decide which module's advice to follow based on the current context. To understand why these networks behave differently, the authors use dynamical systems analysis, probing the "phase space" of the RNNs to find "slow points" and attractors.

The study uses a custom-designed "battery" of navigation tasks based on a discretized 15x15 square arena (inspired by the Morris water maze).

Navigation tasks and agent setup

These tasks include:

  • Basic (BSC): Reach a context-dependent reward location.
  • Hole (HO): Random obstacles are placed in the arena each trial.
  • Bar (BAR): A horizontal or vertical bar is placed randomly.
  • Scale (SC/SX/SY): The arena is stretched in different directions, testing if the agent understands relative landmark positions.
  • Implicit Context (IM): The reward location is signaled by wall color rather than an external signal.

Key Insights and Ideas

  • Inductive Bias = Attractor Landscapes: The most profound insight is that pre-training creates physical "shapes" in the network's activity space.
  • PosNets create a continuous plane attractor—a smooth manifold where the network can "slide" its representation to track $(x, y)$ coordinates.
  • MemNets create discrete fixed points—separate "buckets" of activity that represent different landmarks, allowing for robust memory without needing a coordinate system.
  • The Metric-Topological Trade-off: There is a clear performance split. PosNets are superior at "metric" tasks (like avoiding random bars) because they maintain a sense of space. MemNets are superior at "topological" tasks (like scaling) because they focus on the relationship between landmarks and rewards, regardless of the exact distance.
  • Modularity Wins: A single RNN often struggles to be both a map-reader and a landmark-rememberer. A modular architecture that combines both types of dynamics significantly outperforms any single-module network.

Key Results

The two-stage approach was significantly more data-efficient than standard end-to-end training, which often failed entirely to learn the topological (scaling) tasks that the pre-trained MemNets solved easily.

Network architecture and performance heatmaps across tasks
  • Performance Split: As shown in the performance heatmaps, PosNets achieved high scores on metric tasks like "Bar" but struggled on "Scale" tasks. Conversely, MemNets excelled at "Scale" but failed at "Bar."
  • PCA Analysis: Performing Principal Component Analysis on the performance matrix revealed that 79% of the variance in how networks performed across tasks could be explained by two factors: task difficulty and the metric-topological axis.
  • Visualization of Dynamics: The authors successfully visualized the "Slow Manifold." They showed that in PosNets, position is smoothly encoded on a manifold, whereas in MemNets, the same space is occupied by four distinct clusters representing the four walls.

Limitations and Future Work

  • Toy Environments: The study is limited to a 2D grid-world with very simple proximal sensory inputs (only seeing 8 neighboring cells). It remains to be seen how these attractor dynamics scale to complex, high-dimensional visual environments (like 3D simulators or the real world).
  • Fixed Connectivity: In the second stage, internal connectivity is frozen. Future work could explore "continual learning" where the attractors are allowed to drift or refine during the task-specific phase.
  • Other Attractor Types: The paper focuses on plane and point attractors. The authors suggest that other structures, like line attractors (useful for timing or integration), could serve as inductive biases for different domains beyond navigation.

Summary updated

Abstract

Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map. The precise form of this representation is often considered to be a metric representation of space. An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks. Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment. To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output. We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state. These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities. Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -- which can be shaped by pre-training and analyzed using dynamical systems methods. Furthermore, we demonstrate that non-metric representations are useful for navigation tasks.
BibTeX
@inproceedings{xu2020implementing,
  author = {Tie Xu and Omri Barak},
  title = {{Implementing Inductive bias for different navigation tasks through diverse RNN attrractors}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=Y8D2A_-_Imn},
}
Figure