Abstract
3D object detection from monocular video is a critical yet challenging task for autonomous driving. While existing methods leverage temporal information through filtering, feature fusion, or pseudo-stereo, they often struggle with moving objects or lack explicit geometric constraints. This paper introduces BA-Det, a novel framework that incorporates a learnable, object-centric version of Bundle Adjustment (BA). By optimizing object poses across multiple frames using dense featuremetric correspondences, BA-Det treats moving objects as static relative to their own reference frames. Evaluated on the Waymo Open Dataset, BA-Det achieves state-of-the-art performance, particularly demonstrating a 3x improvement in detecting distant objects compared to single-frame baselines.
Introduction
Monocular 3D object detection is a massive challenge for autonomous systems. While single-frame detectors have seen significant improvements, they inherently ignore the rich temporal context available in video streams. Existing video-based approaches typically fall into three categories: tracking-based filtering, temporal Bird’s Eye View (BEV) feature fusion, and stereo-from-video. However, these methods often fail to enforce explicit geometric constraints or struggle with moving objects that violate standard epipolar geometry.
BA-Det addresses these limitations by introducing a two-stage pipeline that utilizes long-term temporal correspondences to refine object poses for both static and moving targets through a differentiable optimization process.
Methodology
The BA-Det framework consists of two primary stages designed to bridge the gap between deep learning and classical geometric optimization.
Stage 1: Detection and Association
The process begins with a standard monocular 3D detector (e.g., MonoFlex) identifying objects in individual frames. These detections are then linked across time into "tracklets" using a tracker like ImmortalTracker.
Stage 2: Object-Centric Temporal Correspondence Learning (OTCL)
For every object within a tracklet, the model extracts Region of Interest (RoI) features. The OTCL module employs spatial self-attention and temporal cross-attention to learn dense correspondence maps between these features across different frames. This provides the "matches" necessary for geometric refinement.
Featuremetric Object Bundle Adjustment (OBA)
The core innovation is the adaptation of Bundle Adjustment (BA)—a technique traditionally used to minimize reprojection error for camera poses—into an object-centric, learnable format:
- Object-Centricity: By optimizing the object's pose in its own reference frame, moving objects are effectively treated as static relative to themselves. This solves the issue of moving targets violating standard multi-view geometry.
- Featuremetric Loss: Instead of matching raw pixels, the system minimizes the distance between learned feature vectors from the OTCL module. This makes the optimization differentiable and allows the network to learn features optimized for the BA solver.
Experimental Results
The model was rigorously evaluated on the Waymo Open Dataset (WOD), a benchmark for autonomous driving.
Performance Gains
On WOD v1.2, BA-Det achieved 16.60 3D [email protected], a substantial improvement over the MonoFlex baseline (11.70) and outperforming other temporal methods like BEVFormer.
Distant Object Mastery
The most significant impact was observed for distant objects (50m+). In this range, BA-Det's performance was 3x higher than the baseline (3.86 AP vs 1.05 AP), highlighting the stability provided by long-term temporal context (averaging 10-20 frames).
Efficiency
Despite the complexity of the optimization, the inference latency remains manageable at approximately 181.5ms per image, with the BA component itself accounting for only ~19ms.
Conclusion
BA-Det demonstrates that classical geometric optimization can be effectively integrated into deep learning frameworks for 3D video object detection. By shifting the focus to object-centric coordinates and using differentiable featuremetric losses, the model overcomes the hurdles of moving objects and long-range depth estimation. Future work aims to extend this framework to non-rigid entities, such as pedestrians and cyclists, by incorporating mesh-based or skeleton-based geometric constraints.
Summary updated