Introduction
Most modern 3D object detection systems follow a "tracking-by-detection" paradigm. In this setup, the detector is essentially "forgetful": it treats every video frame as a brand-new image, searching the entire 3D space from scratch. Tracking is only added as a post-processing step to link these independent detections.
This approach is both inefficient and counter-intuitive. Humans maintain a "spatial working memory" of their surroundings; we don't re-scan a room every time we blink. Existing AI systems waste computational power searching empty regions and often "lose" objects the moment they are partially hidden (occluded) or move slightly out of frame (truncated) because the detector lacks the temporal context to know where they should be. UDOLO (You Don’t Only Look Once) addresses this by integrating 3D object detection and tracking through a persistent spatial-temporal memory.
The UDOLO Framework
The core of the UDOLO system is its Spatial-Temporal Memory, which consists of two primary components:
- Object Occupancy Map (OOM): A 2D Bird’s Eye View (BEV) map that tracks the likelihood of a location being occupied by an object based on previous frames and ego-motion.
- Object Future States: Predictions of where known objects will be in the next frame, generated by a Kalman Filter.
This memory is integrated into a standard two-stage detector (such as PointRCNN) at two distinct levels:
- Early Integration (OOM-Guided RPN): The Region Proposal Network (RPN) is guided to focus only on areas marked as "high occupancy" or "unobserved" in the OOM. This drastically reduces the number of anchors the system needs to evaluate, saving significant computational time and power.
- Middle Integration (Fusion R-CNN): In the second stage, the system fuses current-frame proposals with the "future state" predictions from memory. This allows the detector to leverage its "recollection" of an object to produce a more accurate bounding box, even if the current visual data is incomplete due to occlusion or truncation.
Experimental Evaluation
The authors evaluated UDOLO on two major benchmarks: ScanNet (indoor RGB-D scans) and KITTI (outdoor LiDAR/Stereo driving data). To test robustness, a specific "occluded" subset of ScanNet was created, containing only heavily hidden or truncated objects.
Key Results
- Robustness to Occlusion: On the ScanNet "occluded" subset, UDOLO improved the mean Average Precision (mAP) from a baseline of 0.91% to 3.23%—a relative improvement of 250%.
- Improved Tracking: On the KITTI dataset, UDOLO significantly outperformed traditional tracking-by-detection baselines in Multi-Object Tracking Accuracy (MOTA), largely by reducing false negatives (missed detections).
- Efficiency and Speed: By pruning the search space through early integration, the system reduced the runtime of the PointRCNN backbone and RPN on KITTI from 42ms to 32ms, a 24% speedup, while maintaining detection accuracy.
Discussion and Future Work
UDOLO represents a shift from a linear pipeline (Detect $\rightarrow$ Track) to a feedback loop (Detect $\leftrightarrow$ Track). By using temporal consistency as a prior rather than just a post-processing step, the system achieves greater stability and efficiency.
However, some limitations remain. The system's performance is tied to the accuracy of ego-motion estimation; if odometry fails, the spatial memory becomes misaligned. Additionally, while the 2D OOM is efficient, complex multi-level environments might eventually require a fully 3D occupancy grid. Future work could also involve adapting this feedback mechanism for faster, one-stage detectors.
Summary updated