Enhancing Embodied Object Detection with Spatial Feature Memory

Nicolas Harvey Chapman, Christopher Lehnert, Will Browne, Feras DayoubWACV 2025

TL;DR

This paper introduces a novel framework that enhances object detection for robots by incorporating a "spatial feature memory." This memory stores and leverages past observations of a scene, leading to more consistent and reliable detections, especially when robots repeatedly encounter the same objects from different viewpoints. The approach significantly improves downstream robotic tasks like semantic mapping and object recall.

What Problem Does This Paper Address?

Robots operating in real-world environments, like homes or warehouses, frequently revisit the same locations and observe the same objects. Traditional object detection systems, often trained on single images or short video clips, struggle with this "embodied" scenario. They can produce inconsistent detections for the same object over time due to varying perspectives, lighting conditions, or partial occlusions. This inconsistency hinders crucial robotic tasks that rely on a stable understanding of the environment, such as building accurate maps or navigating effectively. The paper aims to bridge this gap by developing an object detector that can learn from and exploit its own history of observations within a scene.

Methods

The core innovation is a "spatial feature memory" that acts as a persistent, learned representation of the observed environment. Here's how it works:

  • Memory Update (Writing): When the robot detects an object with high confidence, its features are projected from the 2D image plane into a 3D spatial memory. This projection uses projective geometry and the robot's pose to map the object's location in the 3D world. These projected features are then integrated into the spatial memory, which is essentially a grid-like structure. A counter also tracks how many times each location in the memory has been viewed.
  • Memory Usage (Reading): For subsequent object detection tasks, the system first normalizes the features in the spatial memory based on how often they've been viewed. This helps emphasize frequently observed areas. These normalized memory features are then projected back into the image frame.
  • Feature Fusion: The projected memory features are fused with the features extracted from the current image by the base object detector. This fusion is a weighted sum, allowing the system to balance information from the current observation and its historical memory. The learned weight determines the contribution of each source.
  • End-to-End Training: The entire system, including the spatial memory update and fusion mechanisms, is trained end-to-end using a standard object detection loss. This allows the memory to learn what information is most useful to store and how to best integrate it.

The authors also explore different ways to structure this spatial memory, ultimately favoring one that aggregates features from the shared language-image embedding space (like those from CLIP). This allows the memory to implicitly express confidence in its predictions.

Datasets

The experiments were conducted using simulated and real-world datasets:

  • Simulation:
    • Habitat simulator with Matterport3D scans: Used for training and evaluating the embodied object detection task. This dataset provides semantic labels and RGB-D images from various viewpoints in indoor environments.
  • Replica dataset: Used as an additional test set for evaluating performance in different indoor scenes.
  • Real-World:
    • ScanNet and ScanNet++ datasets: Used to evaluate performance on real-world RGB-D data, accounting for challenges like degraded image quality and imperfect robot pose.
  • Custom robot data: Collected using a mobile robot platform in an office environment, with traversals in different directions to test robustness.

Key Insights and Ideas

  • Embodiment as a Feature: The paper demonstrates that a robot's inherent ability to move and observe its environment over time is a powerful signal that can be explicitly leveraged to improve object detection.
  • Memory for Consistency: A persistent spatial memory is crucial for achieving consistent object detections across repeated observations, overcoming the limitations of single-image detectors.
  • Implicit Confidence: The chosen memory structure, which aggregates features from language-image embeddings, implicitly encodes confidence in both object classification and spatial occupancy. This helps the detector learn when to trust the memory and when to rely more on the current image.
  • Projective Geometry as an Inductive Bias: Using projective geometry for memory read/write operations simplifies the process and reduces the need for complex recurrent architectures, making the system more efficient.
  • Language-Image Pre-training Synergy: The proposed spatial memory can be effectively added to existing language-image pre-trained detectors, enhancing their performance without discarding the benefits of this pre-training.

Key Results

  • Performance Improvement: The proposed spatial feature memory improved object detection performance by up to 4.12 mAP on the Matterport3D dataset and 3.07 mAP on the Replica dataset when added to a fine-tuned DETIC model.
  • Downstream Task Benefits: The approach led to improved performance in downstream tasks such as semantic mapping (improving mean IoU by 3.06 points) and object recall (increasing precision by 0.1).
  • Robustness to Image Quality: The memory provided significant benefits (up to 4.76 mAP) when testing images were intentionally degraded in quality, demonstrating its ability to maintain consistent predictions.
  • Exploration Time Matters: Detection accuracy improved with longer exploration times, showing the value of accumulating more observational data.
  • Real-World Effectiveness: The system demonstrated robust performance on real-world RGB-D data, even with sensor noise and domain shifts.

Limitations and Future Work

  • 3D Geometry and Memory Resolution: While 3D geometry could further disambiguate objects, increasing the spatial resolution of the memory leads to exponential increases in computational cost, making real-time performance a challenge.
  • Dynamic Objects: The current framework assumes objects persist in their previously observed locations. Future work could integrate methods for modeling object dynamics to handle moving objects.
  • Base Detector Flexibility: The authors suggest exploring alternative base detectors beyond the FPN-based CenterNet2, potentially with different architectures or pre-training regimes.
  • Memory Structure: Further research into optimal spatial memory structures and how to best leverage 3D geometry for memory disambiguation is suggested.

Summary updated

BibTeX
@inproceedings{chapman2025enhancing,
  author = {Nicolas Harvey Chapman and Christopher Lehnert and Will Browne and Feras Dayoub},
  title = {{Enhancing Embodied Object Detection with Spatial Feature Memory}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure