MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddings

Victor Rambaud, Salvador Mascarenhas, Yair LakretzarXiv 2025 · DOI · arXiv

Introduction

Current Transformer architectures, while powerful, often struggle with out-of-distribution (OOD) generalization in tasks requiring structural or spatial reasoning. While humans can easily infer shortcuts in a known environment or apply relational rules to novel contexts, standard models like those using Rotary Positional Embeddings (RoPE) tend to rely on statistical shortcuts that fail when sequence lengths or environmental complexities increase. The authors of MapFormer propose that this limitation stems from the lack of a "cognitive map"—an internal representation that disentangles invariant structural relationships ("where") from variable sensory content ("what").

The MapFormer Architecture

MapFormer introduces a novel approach to positional encoding by making it input-dependent and grounded in Lie-group theory. Unlike fixed positional encodings, MapFormer treats actions as operators that transform the coordinate space.

Architectural variants of MapFormer: (a) MapEM (Episodic Memory) and (b) MapWM (Working Memory).

The model features two primary variants:

  1. MapEM (Episodic Memory): This variant explicitly separates structure and content into distinct branches (as seen in Figure 4a). It mimics the hippocampal function of storing specific events within a spatial context, using absolute positional embeddings to maintain this separation.
  2. MapWM (Working Memory): This variant uses relative positional encoding (Figure 4b), where structural information is implicitly represented by rotating neural activity into different subspaces via a rotation matrix.

Parallel Path Integration via Lie Groups

A key technical innovation is the use of Lie-group theory to perform path integration in parallel. In traditional recurrent models, tracking movement requires sequential matrix multiplications, which is computationally expensive. MapFormer overcomes this by performing "path integration" in the Lie algebra space. Actions are treated as infinitesimal rotations that can be summed linearly across a sequence. This linear sum is then mapped back to the Lie group (a rotation matrix) via an exponential map, allowing the model to calculate its relative position at any point in the sequence simultaneously.

Illustration of path integration in spatial (grid) and relational (tree) domains.

As shown in Figure 2, this framework allows the model to treat actions as matrices that transform coordinate space, while observations are treated as vectors within that space. This mathematical distinction enables the model to learn the underlying geometry of a task rather than just memorizing sequences.

Experimental Results

MapFormer was evaluated on several synthetic benchmarks, including 1D to 5D grid navigation and non-commutative tasks.

  • OOD Generalization: On 2D navigation tasks, MapFormer achieved near-perfect (1.0) accuracy on both IID and OOD tests (e.g., longer sequences). In contrast, baseline models like RoPE saw performance drop below 0.40 as sequence length increased.
  • Structural Mastery: The model successfully handled "Selective Copy" tasks and complex navigation where the order of actions matters (non-commutative groups), demonstrating its ability to learn true relational structures.
  • Scaling: MapEM proved more robust than MapWM for larger vocabularies and longer sequences, suggesting that explicit disentanglement of "where" and "what" is a superior architectural choice for complex recall tasks.

Discussion and Future Work

While MapFormer shows remarkable success on synthetic structural tasks, several challenges remain. The current formulation is restricted to causal (decoder-only) Transformers due to the temporal nature of the Lie algebra summation. Additionally, while the model excels at grid-like geometries, extending this to the high-dimensional, non-commutative structures found in natural language or complex visual scenes is a primary goal for future research. Nonetheless, MapFormer provides a promising blueprint for building AI systems with more robust, human-like cognitive mapping capabilities.

Summary updated

BibTeX
@misc{rambaud2025mapformer,
  author = {Victor Rambaud and Salvador Mascarenhas and Yair Lakretz},
  title = {{MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddings}},
  year = {2025},
  eprint = {2511.19279},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.19279},
}
Figure