Continual Learning for Neural Implicit Spatial Mapping

This direction is split into other directions: catastrophic forgetting in neural mapping. Its last version stays readable and citable here.

Version 1 of 177 papers92 conceptsfrozen 2026-10-04

When a robot moves through a new environment, it must build a map to navigate safely. Traditionally, these maps were rigid structures like voxel grids or point clouds. Recently, researchers have shifted toward neural implicit representations, where a neural network—often a Multilayer Perceptron (MLP)—is trained to represent the geometry of a scene as a Continuous Signed Distance Function (SDF) or a Neural Radiance Field (NeRF).

However, these neural maps face a fundamental biological-like hurdle: catastrophic forgetting. Because a robot observes the world as a sequence of images, the neural network is updated incrementally. When the network learns the details of a new room, the gradient updates often overwrite the weights that represented the previous room. This "representational drift" causes the map to dissolve behind the robot as it moves forward.

The Stability-Plasticity Dilemma in Space

The core challenge in online mapping is the stability-plasticity tradeoff. A mapping system must be "plastic" enough to integrate new observations quickly but "stable" enough to preserve what it has already learned.

In iMAP, Edgar Sucar and colleagues demonstrated that a single MLP could represent an entire room in real-time. To fight forgetting, they introduced a replay buffer of keyframes. By periodically re-sampling and training on old data alongside new data, the network maintains a "global" memory. This approach, also explored in Continual Neural Mapping by Zike Yan, treats mapping as an online continual learning task where the "experience" is the stream of sensor data.

Anchoring Knowledge with Hierarchical Representations

While replay buffers help, they scale poorly. As a robot explores an entire building, the number of keyframes required to prevent forgetting grows, eventually exceeding real-time compute limits.

To solve this, NICE-SLAM, developed by Zihan Zhu and Songyou Peng, introduced a hierarchical feature grid. Instead of storing all spatial knowledge in global weights, they distribute it across a multi-level grid. When the robot is in Room A, only the grid cells near Room A are updated. This spatial "anchoring" ensures that updates in one area do not corrupt the representation of distant areas, effectively localizing the learning process.

This hybrid approach—combining explicit spatial structures with implicit neural decoders—is a recurring theme. For instance, HIO-SDF uses a coarse voxel grid to prevent forgetting while a local neural module handles fine details. Similarly, $\nabla$-SDF by Nikolay Atanasov uses a gradient-augmented octree to provide a persistent geometric prior that the neural network can then refine.

From Weights to Gaussians

A more recent shift in the field moves away from MLPs entirely to avoid the forgetting problem at its source. 3D Gaussian Splatting (3DGS) represents the world as a collection of explicit, differentiable "splats" in 3D space.

Systems like SplaTAM and Gaussian-SLAM use these Gaussians as an explicit memory. Because each Gaussian is a discrete entity with a specific location, adding new information simply means adding new Gaussians or optimizing local ones. This bypasses the catastrophic forgetting inherent in shared neural weights, allowing for photorealistic reconstruction and high-speed tracking without the risk of "overwriting" the past.

Lifelong Autonomy and Place Recognition

Beyond just geometry, robots need to recognize where they are over long timescales. Peng Yin and others have framed this as General Place Recognition (GPR). In long-term navigation, the environment itself changes—seasons pass, or furniture moves.

Standard models suffer from "Belief Inertia" or total forgetting when exposed to these changes. Recent work, such as the KDF framework by Xufei Wang and Junqiao Zhao, uses knowledge distillation to fuse old and new spatial memories. By preserving the "ranking order" of how locations are recognized, these systems allow robots to adapt to new environments without losing the ability to recognize the original state of a location.

Why It Works

The transition from purely implicit (global MLP) to hybrid (grids/octrees) and eventually explicit (Gaussians) representations works because it decouples the learning of local details from the storage of global structure.

  1. Locality: By restricting weight updates to specific spatial regions, systems prevent interference between distant observations.
  2. Replay: By maintaining a small, representative set of past data, systems provide a "constant" signal that prevents the network from drifting.
  3. Explicit Anchors: By using voxels or Gaussians as a backbone, the system provides a persistent "skeleton" that the neural network can flesh out without losing the overall shape of the world.

As Xingyu Chen suggests in TTT3R, treating 3D reconstruction as a form of Test-Time Training—where the model's internal state is treated as "fast weights" updated on the fly—allows robots to maintain a constant memory footprint while continuously refining their understanding of the world.

Papers in this version

Showing 60 of 77 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continual_learning_for_neural_implicit_s_a6fbe9a6,
  title = {Continual Learning for Neural Implicit Spatial Mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continual-learning-for-neural-implicit-spatial-mapping/rdr_a6fbe9a6c358412594f7b14bc9223a9c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_continual_learning_for_neural_implicit_s_a6fbe9a6_v1,
  title = {Continual Learning for Neural Implicit Spatial Mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continual-learning-for-neural-implicit-spatial-mapping/rdr_a6fbe9a6c358412594f7b14bc9223a9c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}