You Don't Only Look Once: Constructing Spatial-Temporal Memory for Integrated 3D Object Detection and Tracking

Jiaming Sun, Yiming Xie, Siyu Zhang, Linghao Chen, Guofeng Zhang, Hujun Bao, Xiaowei Zhou

Introduction

Most modern 3D object detection systems follow a "tracking-by-detection" paradigm. In this setup, the detector is essentially "forgetful": it treats every video frame as a brand-new image, searching the entire 3D space from scratch. Tracking is only added as a post-processing step to link these independent detections.

This approach is both inefficient and counter-intuitive. Humans maintain a "spatial working memory" of their surroundings; we don't re-scan a room every time we blink. Existing AI systems waste computational power searching empty regions and often "lose" objects the moment they are partially hidden (occluded) or move slightly out of frame (truncated) because the detector lacks the temporal context to know where they should be. UDOLO (You Don’t Only Look Once) addresses this by integrating 3D object detection and tracking through a persistent spatial-temporal memory.

The UDOLO Framework

The core of the UDOLO system is its Spatial-Temporal Memory, which consists of two primary components:

  1. Object Occupancy Map (OOM): A 2D Bird’s Eye View (BEV) map that tracks the likelihood of a location being occupied by an object based on previous frames and ego-motion.
  2. Object Future States: Predictions of where known objects will be in the next frame, generated by a Kalman Filter.
The UDOLO architecture integrates spatial-temporal memory from previous frames (Back-end) into the current frame's detection pipeline (Front-end) via early and middle integration.

This memory is integrated into a standard two-stage detector (such as PointRCNN) at two distinct levels:

  • Early Integration (OOM-Guided RPN): The Region Proposal Network (RPN) is guided to focus only on areas marked as "high occupancy" or "unobserved" in the OOM. This drastically reduces the number of anchors the system needs to evaluate, saving significant computational time and power.
  • Middle Integration (Fusion R-CNN): In the second stage, the system fuses current-frame proposals with the "future state" predictions from memory. This allows the detector to leverage its "recollection" of an object to produce a more accurate bounding box, even if the current visual data is incomplete due to occlusion or truncation.

Experimental Evaluation

The authors evaluated UDOLO on two major benchmarks: ScanNet (indoor RGB-D scans) and KITTI (outdoor LiDAR/Stereo driving data). To test robustness, a specific "occluded" subset of ScanNet was created, containing only heavily hidden or truncated objects.

Key Results

  • Robustness to Occlusion: On the ScanNet "occluded" subset, UDOLO improved the mean Average Precision (mAP) from a baseline of 0.91% to 3.23%—a relative improvement of 250%.
  • Improved Tracking: On the KITTI dataset, UDOLO significantly outperformed traditional tracking-by-detection baselines in Multi-Object Tracking Accuracy (MOTA), largely by reducing false negatives (missed detections).
  • Efficiency and Speed: By pruning the search space through early integration, the system reduced the runtime of the PointRCNN backbone and RPN on KITTI from 42ms to 32ms, a 24% speedup, while maintaining detection accuracy.
Comparison of Average Precision (AP) vs. inference time. UDOLO ("ours") maintains higher accuracy at significantly lower latencies compared to the PRCNN baseline.

Discussion and Future Work

UDOLO represents a shift from a linear pipeline (Detect $\rightarrow$ Track) to a feedback loop (Detect $\leftrightarrow$ Track). By using temporal consistency as a prior rather than just a post-processing step, the system achieves greater stability and efficiency.

However, some limitations remain. The system's performance is tied to the accuracy of ego-motion estimation; if odometry fails, the spatial memory becomes misaligned. Additionally, while the 2D OOM is efficient, complex multi-level environments might eventually require a fully 3D occupancy grid. Future work could also involve adapting this feedback mechanism for faster, one-stage detectors.

Summary updated

Abstract

Humans are able to continuously detect and track surrounding objects by constructing a spatial-temporal memory of the objects when looking around. In contrast, 3D object detectors in existing tracking-by-detection systems often search for objects in every new video frame from scratch, without fully leveraging memory from previous detection results. In this work, we propose a novel system for integrated 3D object detection and tracking, which uses a dynamic object occupancy map and previous object states as spatial-temporal memory to assist object detection in future frames. This memory, together with the ego-motion from back-end odometry, guides the detector to achieve more efficient object proposal generation and more accurate object state estimation. The experiments demonstrate the effectiveness of the proposed system and its performance on the ScanNet and KITTI datasets. Moreover, the proposed system produces stable bounding boxes and pose trajectories over time, while being able to handle occluded and truncated objects. Code is available at the project page: https://zju3dv.github.io/UDOLO.
BibTeX
@inproceedings{sun2021you,
  author = {Jiaming Sun and Yiming Xie and Siyu Zhang and Linghao Chen and Guofeng Zhang and Hujun Bao and Xiaowei Zhou},
  title = {{You Don't Only Look Once: Constructing Spatial-Temporal Memory for Integrated 3D Object Detection and Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00317},
}
Figure