TempBEV: The Best of Both Worlds in Temporal Perception
TL;DR
TempBEV is a novel encoder for autonomous driving that combines temporal aggregation in both high-resolution image space and top-down Bird’s-Eye View (BEV) space. By leveraging the complementary strengths of these two representations—precise short-term motion cues from images and long-term consistency from BEV—it achieves significant performance gains in 3D object detection and map segmentation.
What Problem Does This Paper Address?
In autonomous driving, monocular (camera-only) systems are cost-effective but struggle with 3D perception because a single 2D image lacks explicit depth and velocity information. To fix this, researchers use temporal aggregation: combining information from multiple frames over time to "triangulate" depth (temporal stereo) and estimate object motion.
Current state-of-the-art models typically perform this aggregation in one of two places:
- Image Space: High resolution, but ego-motion (the car's own movement) makes finding correspondences across frames difficult.
- BEV Space: A top-down view where ego-motion is easy to cancel out, but the representation is often low-resolution and suffers from "lifting" errors (uncertainty in the 2D-to-3D conversion).
The authors identify a gap: most models pick one or the other, missing out on the unique advantages of the alternative representation.
Methods
The authors propose TempBEV, which integrates temporal information from both latent spaces simultaneously.
- Image Space Aggregation (Short-term): The model uses a "temporal stereo encoder" based on the FlowNet architecture. It takes pairs of consecutive images and extracts features related to optical flow. This captures precise, high-resolution motion cues that are often lost when projecting to BEV.
- BEV Space Aggregation (Long-term): Building on the popular BEVFormer architecture, TempBEV uses a recurrent mechanism. It maintains a "memory" of the environment in BEV space, using temporal self-attention to update the current state with information from previous time steps. This is ideal for maintaining static map elements and handling occlusions over longer periods.
- Synergistic Fusion: The features from the image-space stereo encoder are "lifted" into 3D and projected into the BEV grid, where they are fused with the recurrent BEV features.
Datasets
The model was evaluated on the nuScenes dataset, a large-scale autonomous driving benchmark featuring 1,000 scenes in Boston and Singapore. It includes data from 6 cameras, LiDAR, and Radar, though this paper focuses on the camera-only track. The authors used the standard metrics for 3D object detection (NDS, mAP) and BEV segmentation (IoU for Road, Lane, and Pedestrian Crossings).
Key Insights and Ideas
- Complementary Strengths: The "aha moment" is the realization that image space and BEV space aren't redundant; they are complementary. Image space provides the "sharpness" needed for dynamic object detection, while BEV provides the "stability" needed for static map elements.
- Transfer Learning from Optical Flow: One of the most interesting findings is that pre-training the image-space encoder on a dedicated optical flow dataset (Flying Chairs) significantly boosts 3D detection performance. This suggests that learning to "see" motion in 2D is a powerful prerequisite for understanding 3D environments.
- Synergy over Scaling: The ablation studies show that adding image-space aggregation to a BEV-only baseline provides a much larger performance boost than simply increasing the number of parameters or layers in the BEV encoder alone.
Key Results
TempBEV was compared against a strong BEVFormer baseline and several other temporal operators (like convolution and max pooling):
- 3D Detection: TempBEV achieved 51.31% NDS and 41.26% mAP, an improvement of +1.06 pts and +1.44 pts respectively over the BEVFormer baseline.
- BEV Segmentation: The model showed significant gains in mapping static elements, particularly in detecting pedestrian crossings (+1.85 pts IoU) and lanes (+1.12 pts IoU).
- Efficiency: While adding the image-space encoder increases the parameter count by about 5.3%, it provides a performance jump that justifies the slightly lower inference speed (1.94 FPS vs 2.18 FPS on an NVIDIA A6000).
Limitations and Future Work
- Computational Overhead: The dual-aggregation approach is more computationally expensive than single-space methods. The authors note a ~11% decrease in FPS.
- Camera-Specific Patterns: The authors experimented with "camera-specific" encoders (training different weights for front vs. side cameras) but found that the nuScenes dataset wasn't large enough to provide sufficient guidance for this, leading to mixed results.
- Future Directions: The authors suggest that using more sophisticated, modern optical flow architectures or integrating these insights into even newer "foundation models" for driving could yield further improvements.
Summary updated