Egocentric Spatial Memory

Mengmi Zhang, Keng Teck Ma, Shih-Cheng Yen, Joo Hwee Lim, Qi Zhao, Jiashi Feng

TL;DR

This paper introduces Egocentric Spatial Memory Network (ESMN), a novel deep learning architecture that enables robots to build accurate 2D global maps from their own first-person perspective. By integrating egocentric views with an external memory system, ESMN can effectively handle long-term exploration, loop closure detection, and map correction, even with noisy motion data.

What Problem Does This Paper Address?

Navigating and mapping complex environments is a fundamental challenge for robots. While traditional methods exist, building a comprehensive, accurate, and robust spatial memory from a robot's egocentric viewpoint, especially over long durations and in dynamic conditions, remains difficult. Existing deep learning approaches often struggle with continuous actions, long-term dependencies, and the need for precise loop closure (recognizing when a robot has returned to a previously visited location) to correct accumulated errors. This paper aims to bridge this gap by proposing an end-to-end deep neural network that mimics aspects of biological spatial memory systems to create a dynamic, top-down global map.

Methods

The core of the proposed solution is the Egocentric Spatial Memory Network (ESMN), a unified deep neural network architecture inspired by neurophysiological findings on spatial navigation cells in the brain. ESMN comprises four key modules:

  • Head Direction Unit (HDU): This module simplifies the robot's motion sensors, providing noise-free egomotion (changes in position and orientation) as input.
  • Boundary Vector Unit (BVU): Acting as a local mapper, BVU uses a recurrent neural network (RNN) to process egocentric camera views and predict local free-space occupancy grids. It integrates past spatial information with current observations to maintain a short-term, high-resolution local map. A spatial transformer network is used to align past information with the current egocentric coordinate system based on egomotion.
  • Place Unit (PU): This module is responsible for place encoding. It learns discriminative representations of visited places using a 2D-CNN trained with a triplet loss. This allows the system to recognize previously visited locations, crucial for loop closure, independent of the agent's orientation.
  • Grid Unit (GU): This module serves as the global mapper and incorporates an external memory system. It integrates local maps from BVU over time to build a global map and stores discriminative representations of visited places. This external memory is key to handling long-term exploration and correcting mapping errors through loop closure.

The system is designed to work with continuous actions and states, making it more suitable for real-world robotic control.

Datasets

The paper evaluates ESMN on two types of environments:

  • 3D Virtual Mazes: These environments are used for initial testing and feature complex geometries and textures.
  • Realistic Indoor Environments: The authors utilize the 2D3DS dataset, which provides real-world indoor scene data collected using Matterport Cameras. This dataset offers large-scale environments with rich semantic information.

Ground truth for 2D top-view environments was obtained using a 2D laser scanner attached to the agent.

Key Insights and Ideas

  • Neuro-inspired Architecture: Mimicking biological spatial memory systems (head direction, boundary vector, place, and grid cells) with neural network modules provides a powerful framework for egocentric spatial memory.
  • External Memory for Long-Term Mapping: The integration of an external memory module is critical for overcoming the limitations of short-term RNNs and enabling robust global mapping over extended periods and large environments.
  • Loop Closure for Map Correction: The ability to detect loop closure (recognizing revisited places) is not just for navigation but is actively used to correct accumulated errors in the global map, significantly improving its accuracy.
  • End-to-End Learning with Ground Truth Egomotion: The framework is designed to learn mapping directly from visual input and egomotion, simplifying the pipeline compared to traditional SLAM systems that often require explicit sensor fusion and state estimation.

Key Results

  • Accurate Local Mapping: ESMN demonstrates accurate prediction of free space in local maps, even in complex indoor settings with narrow corridors. Quantitative analysis shows it outperforms baselines like BiClassi and Chance in terms of Mean Squared Error (MSE), Correlation, and Mutual Information.
  • Robust Loop Closure Detection: The Place Unit effectively encodes visited places, enabling accurate loop closure detection. The precision-recall curve shows ESMN significantly outperforms bag-of-words (SIFT) and pixel-wise comparison methods.
  • Improved Global Mapping: The full ESMN architecture (HDU + BVU + PU + GU) achieves superior global map performance compared to ablated models and baselines like 3D-CNN and LSTM Direct. At t=448, it shows a decrease of 0.03 in MSE, an increase of 0.17 in correlation, and 0.09 in MI compared to a 3D-CNN baseline.
  • Map Correction Benefits: When loop closure is detected and map correction is applied, the global map becomes structurally more accurate, as visually demonstrated and supported by quantitative improvements in correlation and MI, even if MSE differences are small.

Limitations and Future Work

  • Fixed External Memory Size: The Grid Unit's external memory has a fixed size, which could be a limitation for robots operating in extremely large or continuously expanding environments.
  • Supervised Approach: The current discussion focuses on a supervised approach for occupancy grid map construction.
  • Future Directions:
    • Developing a global map with a non-expandable memory size for large-scale missions.
  • Integrating the mapping framework with downstream path-planning modules, potentially using reinforcement learning for goal-directed navigation.
  • Applying the framework to a mobile robotic platform for real-life navigation tasks, allowing it to interact with a control system.

Summary updated

BibTeX
@inproceedings{zhang2018egocentric,
  author = {Mengmi Zhang and Keng Teck Ma and Shih-Cheng Yen and Joo Hwee Lim and Qi Zhao and Jiashi Feng},
  title = {{Egocentric Spatial Memory}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  doi = {10.1109/iros.2018.8593435},
}

Loading related content...

Figure