End-to-End Egospheric Spatial Memory

Daniel James Lenton, Stephen James, Ronald Clark, Andrew J. Davison

Spatial memory—the ability to remember and recall specific locations and objects—is central to an autonomous agent's ability to carry out tasks in real-world environments. However, standard neural network memory modules, such as LSTMs or Neural Turing Machines, struggle with spatial reasoning. They treat memory as a flat vector of numbers, forcing the network to "learn" the laws of 3D geometry from scratch through trial and error.

Conversely, traditional robotic mapping (SLAM) excels at geometry but is usually a separate, non-differentiable system that doesn't integrate smoothly with end-to-end deep learning. To bridge this gap, researchers have introduced Egospheric Spatial Memory (ESM), a differentiable, parameter-free memory module that allows embodied agents to store 3D spatial information in a spherical map centered on themselves.

How Egospheric Spatial Memory Works

The core of ESM is a 2.5D spherical image (an "ego-sphere") that surrounds the agent. Each "pixel" in this memory stores radial depth and high-dimensional features learned by a Convolutional Neural Network (CNN).

The internal operations of the Egospheric Spatial Memory (ESM) module, showing the forward warp, quantization, and fusion steps.

The module operates similarly to an Extended Kalman Filter (EKF) through three main steps:

  1. Motion Step: As the agent moves, the previous memory is "warped" (re-projected) to account for the change in pose.
  2. Observation Step: New visual data from the agent's cameras are projected into the spherical coordinate system.
  3. Update Step: The predicted state (from the motion step) and the new observation are fused using a Kalman gain, which accounts for uncertainty.

Unlike most memory modules, ESM is parameter-free. It doesn't have weights that need training; the "logic" of the memory is hard-coded 3D geometry. However, because the projection and fusion operations are fully differentiable, the CNNs before the memory can learn which features are most useful to store, and the CNNs after the memory can learn how to translate that map into actions.

Experimental Setup

The authors evaluated ESM across several challenging domains to test its versatility and performance:

  • Drone Reacher (DR) & Manipulator Reacher (MR): Simulated environments where a drone or a robotic arm must reach multiple colored or shaped targets in a specific order.
  • ScanNet: A large-scale real-world dataset of 3D indoor office and home environments, used to test ESM's ability to perform semantic segmentation (labeling objects like chairs, tables, and walls) by fusing multiple views over time.
Simulated environments used for evaluation: Drone Reacher (left) and Manipulator Reacher (right).

Key Insights and Results

By using explicit geometric projections instead of "black-box" recurrence, ESM significantly improves learning efficiency and performance on complex navigation and manipulation tasks.

  • Geometry as an Inductive Bias: Forcing the memory to follow the rules of 3D space provides a powerful "hint" to the network. This leads to much faster training and better final performance than "blind" recurrent networks. In the Drone Reacher task, ESM reached an average of 20.8 targets per episode, nearly matching the "Expert" policy (21.3) and far outperforming the LSTM baseline (12.7).
  • Interpretability: Because the memory is a physical map, researchers can visualize exactly what the robot "remembers." If a target goes out of view, its representation visibly persists in the ego-sphere.
  • Camera Generalization: ESM-based policies can generalize between different camera types. A policy trained on a fixed body-mounted camera could still function when switched to a "free-form" flying camera—a feat the LSTM baselines failed at entirely.
  • Seamless Modality Fusion: Because the memory is spatially grounded, it is easy to combine learned features with non-learned ones. For instance, the authors successfully added a hard-coded obstacle avoidance "force" to a learned navigation policy.
  • Efficiency and Semantic Mapping: Despite the complex geometry, ESM is highly efficient, running at hundreds of frames per second (FPS) on a standard GPU. On the ScanNet dataset, ESM improved semantic segmentation accuracy by effectively "smoothing" predictions over time and integrating wider context that a single monocular frame would miss.

Limitations and Future Work

While powerful, ESM has a few limitations. It relies heavily on accurate incremental pose measurements (odometry); if the robot's sense of its own movement is noisy, the memory will become "blurry." Additionally, the current warping logic assumes a static environment. If objects in the scene move independently, they might leave "ghosts" or smears in the memory.

Future work will likely focus on extending ESM to handle more significant uncertainty in real-world measurements and exploring mechanisms to manage dynamic, moving objects within the spherical memory framework.

Summary updated

Abstract

Spatial memory, or the ability to remember and recall specific locations and objects, is central to autonomous agents' ability to carry out tasks in real environments. However, most existing artificial memory modules are not very adept at storing spatial information. We propose a parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent, enabling expressive 3D representations. ESM can be trained end-to-end via either imitation or reinforcement learning, and improves both training efficiency and final performance against other memory baselines on both drone and manipulator visuomotor control tasks. The explicit egocentric geometry also enables us to seamlessly combine the learned controller with other non-learned modalities, such as local obstacle avoidance. We further show applications to semantic segmentation on the ScanNet dataset, where ESM naturally combines image-level and map-level inference modalities. Through our broad set of experiments, we show that ESM provides a general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures. Implementation at: https://github.com/ivy-dl/memory.
BibTeX
@inproceedings{lenton2021end,
  author = {Daniel James Lenton and Stephen James and Ronald Clark and Andrew J. Davison},
  title = {{End-to-End Egospheric Spatial Memory}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=UetEcbi4eIf},
}
Figure