Abstract
Traditional 3D object detection methods often process video sequences as a series of independent frames, failing to exploit the rich temporal context inherent in streaming data. This paper introduces Dual-way Object Detection and Tracking (DODT), a framework designed to bridge this gap. DODT processes only specific "keyframes" through a dual-way network and propagates these results to intermediate frames using a motion-based interpolation (MoI) algorithm. This approach achieves a Multiple Object Tracking Accuracy (MOTA) of 76.68% and maintains high efficiency, reaching up to 76.9 FPS.
Introduction
Current 3D object detection models frequently ignore temporal information, leading to several drawbacks. Single-frame analysis often lacks the context needed to identify occluded or truncated objects that might be visible in adjacent frames. Furthermore, running heavy 3D detection on every frame is computationally expensive and often redundant, as objects move gradually. This frame-by-frame approach also tends to result in temporal inconsistency, or "flickering," in detections.
While 2D video object detection has matured, 3D streaming data—combining sparse LiDAR point clouds and RGB images—presents unique challenges in encoding temporal features without excessive computational overhead.
Methodology
The DODT framework comprises three primary components: the Shared Region Proposal Network (RPN), the Temporal Module, and Motion-based Interpolation (MoI).
Shared RPN
Instead of generating proposals for each frame independently, the Shared RPN creates a set of 3D proposals shared by two adjacent keyframes. It integrates Bird’s Eye View (BEV) feature maps across multiple frames to generate robust proposals, particularly for moving objects. This fusion of BEV point cloud data with RGB image features allows the model to benefit from both precise spatial localization and rich appearance details.
Temporal Module
This module performs cross-correlation at the proposal level rather than the pixel level. By comparing feature crops from two keyframes, it predicts "co-occurrence" (the presence of the same object in both frames) and "offsets" (the object's displacement between keyframes). Proposal-level correlation is significantly more efficient and flexible than correlating entire feature maps.
Motion-based Interpolation (MoI)
DODT performs full detection only on keyframes separated by a temporal stride $\tau$. The MoI algorithm uses predicted offsets and a constant-velocity motion model to interpolate object positions for all intermediate frames, drastically increasing processing speed.
Experimental Results
The model was evaluated on the KITTI Object Tracking Benchmark. To ensure rigorous evaluation, the authors manually refined KITTI labels where they were found to be incomplete, ensuring a more accurate assessment of the model's predictive power.
Performance and Efficiency
DODT achieved a 76.68% MOTA and 81.65% MOTP (Multiple Object Tracking Precision). Notably, compared to the baseline AVOD detector, DODT improved Average Precision (AP) in the "Hard" category by nearly 10% (from 56.13% to 68.75% at IoU=0.7).
The system demonstrates high efficiency. While detection branches require approximately 175ms, interpolation is nearly instantaneous. With a temporal stride of $\tau=3$, the system achieves a throughput of 76.9 FPS. Interestingly, a larger stride was found to improve the detection of occluded objects by providing a wider temporal window.
Conclusion and Future Work
DODT offers a robust and efficient solution for 3D object detection and tracking in streaming data by leveraging proposal-level temporal correlation. Future work will focus on expanding the model to handle more diverse object classes, such as pedestrians and cyclists, and refining the motion model to better accommodate complex urban driving scenarios where constant-velocity assumptions may fail. Additionally, reducing the inherent latency of the "near online" interpolation approach remains a priority for time-critical autonomous applications.
Summary updated