3D Video Object Detection with Learnable Object-Centric Global Optimization

Jiawei He, Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang

Abstract

3D object detection from monocular video is a critical yet challenging task for autonomous driving. While existing methods leverage temporal information through filtering, feature fusion, or pseudo-stereo, they often struggle with moving objects or lack explicit geometric constraints. This paper introduces BA-Det, a novel framework that incorporates a learnable, object-centric version of Bundle Adjustment (BA). By optimizing object poses across multiple frames using dense featuremetric correspondences, BA-Det treats moving objects as static relative to their own reference frames. Evaluated on the Waymo Open Dataset, BA-Det achieves state-of-the-art performance, particularly demonstrating a 3x improvement in detecting distant objects compared to single-frame baselines.

Introduction

Monocular 3D object detection is a massive challenge for autonomous systems. While single-frame detectors have seen significant improvements, they inherently ignore the rich temporal context available in video streams. Existing video-based approaches typically fall into three categories: tracking-based filtering, temporal Bird’s Eye View (BEV) feature fusion, and stereo-from-video. However, these methods often fail to enforce explicit geometric constraints or struggle with moving objects that violate standard epipolar geometry.

Comparison of temporal modeling paradigms in 3D object detection.

BA-Det addresses these limitations by introducing a two-stage pipeline that utilizes long-term temporal correspondences to refine object poses for both static and moving targets through a differentiable optimization process.

Methodology

The BA-Det framework consists of two primary stages designed to bridge the gap between deep learning and classical geometric optimization.

Stage 1: Detection and Association

The process begins with a standard monocular 3D detector (e.g., MonoFlex) identifying objects in individual frames. These detections are then linked across time into "tracklets" using a tracker like ImmortalTracker.

Stage 2: Object-Centric Temporal Correspondence Learning (OTCL)

For every object within a tracklet, the model extracts Region of Interest (RoI) features. The OTCL module employs spatial self-attention and temporal cross-attention to learn dense correspondence maps between these features across different frames. This provides the "matches" necessary for geometric refinement.

Overview of the BA-Det framework, illustrating the pipeline from image input to the OTCL and OBA modules.

Featuremetric Object Bundle Adjustment (OBA)

The core innovation is the adaptation of Bundle Adjustment (BA)—a technique traditionally used to minimize reprojection error for camera poses—into an object-centric, learnable format:

  • Object-Centricity: By optimizing the object's pose in its own reference frame, moving objects are effectively treated as static relative to themselves. This solves the issue of moving targets violating standard multi-view geometry.
  • Featuremetric Loss: Instead of matching raw pixels, the system minimizes the distance between learned feature vectors from the OTCL module. This makes the optimization differentiable and allows the network to learn features optimized for the BA solver.

Experimental Results

The model was rigorously evaluated on the Waymo Open Dataset (WOD), a benchmark for autonomous driving.

Performance Gains

On WOD v1.2, BA-Det achieved 16.60 3D [email protected], a substantial improvement over the MonoFlex baseline (11.70) and outperforming other temporal methods like BEVFormer.

Distant Object Mastery

The most significant impact was observed for distant objects (50m+). In this range, BA-Det's performance was 3x higher than the baseline (3.86 AP vs 1.05 AP), highlighting the stability provided by long-term temporal context (averaging 10-20 frames).

Efficiency

Despite the complexity of the optimization, the inference latency remains manageable at approximately 181.5ms per image, with the BA component itself accounting for only ~19ms.

Conclusion

BA-Det demonstrates that classical geometric optimization can be effectively integrated into deep learning frameworks for 3D video object detection. By shifting the focus to object-centric coordinates and using differentiable featuremetric losses, the model overcomes the hurdles of moving objects and long-range depth estimation. Future work aims to extend this framework to non-rigid entities, such as pedestrians and cyclists, by incorporating mesh-based or skeleton-based geometric constraints.

Summary updated

Abstract

We explore long-term temporal visual correspondence-based optimization for 3D video object detection in this work. Visual correspondence refers to one-to-one mappings for pixels across multiple images. Correspondence-based optimization is the cornerstone for 3D scene reconstruction but is less studied in 3D video object detection, because moving objects violate multi-view geometry constraints and are treated as outliers during scene reconstruction. We address this issue by treating objects as first-class citizens during correspondence-based optimization. In this work, we propose BA-Det, an end-to-end optimizable object detector with object-centric temporal correspondence learning and featuremetric object bundle adjustment. Empirically, we verify the effectiveness and efficiency of BA-Det for multiple baseline 3D detectors under various setups. Our BA-Det achieves SOTA performance on the large-scale Waymo Open Dataset (WOD) with only marginal computation cost. Our code is available at https://github.com/jiaweihe1996/BA-Det.
BibTeX
@inproceedings{he2023d,
  author = {Jiawei He and Yuntao Chen and Naiyan Wang and Zhaoxiang Zhang},
  title = {{3D Video Object Detection with Learnable Object-Centric Global Optimization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00494},
}
Figure