TempBEV: Improving Learned BEV Encoders with Combined Image and BEV Space Temporal Aggregation

Thomas Monninger, Vandana Dokkadi, Md Zafar Anwar, Steffen Staab

TempBEV: The Best of Both Worlds in Temporal Perception

TL;DR

TempBEV is a novel encoder for autonomous driving that combines temporal aggregation in both high-resolution image space and top-down Bird’s-Eye View (BEV) space. By leveraging the complementary strengths of these two representations—precise short-term motion cues from images and long-term consistency from BEV—it achieves significant performance gains in 3D object detection and map segmentation.

What Problem Does This Paper Address?

In autonomous driving, monocular (camera-only) systems are cost-effective but struggle with 3D perception because a single 2D image lacks explicit depth and velocity information. To fix this, researchers use temporal aggregation: combining information from multiple frames over time to "triangulate" depth (temporal stereo) and estimate object motion.

Current state-of-the-art models typically perform this aggregation in one of two places:

  1. Image Space: High resolution, but ego-motion (the car's own movement) makes finding correspondences across frames difficult.
  2. BEV Space: A top-down view where ego-motion is easy to cancel out, but the representation is often low-resolution and suffers from "lifting" errors (uncertainty in the 2D-to-3D conversion).

The authors identify a gap: most models pick one or the other, missing out on the unique advantages of the alternative representation.

Methods

The authors propose TempBEV, which integrates temporal information from both latent spaces simultaneously.

  • Image Space Aggregation (Short-term): The model uses a "temporal stereo encoder" based on the FlowNet architecture. It takes pairs of consecutive images and extracts features related to optical flow. This captures precise, high-resolution motion cues that are often lost when projecting to BEV.
  • BEV Space Aggregation (Long-term): Building on the popular BEVFormer architecture, TempBEV uses a recurrent mechanism. It maintains a "memory" of the environment in BEV space, using temporal self-attention to update the current state with information from previous time steps. This is ideal for maintaining static map elements and handling occlusions over longer periods.
  • Synergistic Fusion: The features from the image-space stereo encoder are "lifted" into 3D and projected into the BEV grid, where they are fused with the recurrent BEV features.

Datasets

The model was evaluated on the nuScenes dataset, a large-scale autonomous driving benchmark featuring 1,000 scenes in Boston and Singapore. It includes data from 6 cameras, LiDAR, and Radar, though this paper focuses on the camera-only track. The authors used the standard metrics for 3D object detection (NDS, mAP) and BEV segmentation (IoU for Road, Lane, and Pedestrian Crossings).

Key Insights and Ideas

  • Complementary Strengths: The "aha moment" is the realization that image space and BEV space aren't redundant; they are complementary. Image space provides the "sharpness" needed for dynamic object detection, while BEV provides the "stability" needed for static map elements.
  • Transfer Learning from Optical Flow: One of the most interesting findings is that pre-training the image-space encoder on a dedicated optical flow dataset (Flying Chairs) significantly boosts 3D detection performance. This suggests that learning to "see" motion in 2D is a powerful prerequisite for understanding 3D environments.
  • Synergy over Scaling: The ablation studies show that adding image-space aggregation to a BEV-only baseline provides a much larger performance boost than simply increasing the number of parameters or layers in the BEV encoder alone.

Key Results

TempBEV was compared against a strong BEVFormer baseline and several other temporal operators (like convolution and max pooling):

  • 3D Detection: TempBEV achieved 51.31% NDS and 41.26% mAP, an improvement of +1.06 pts and +1.44 pts respectively over the BEVFormer baseline.
  • BEV Segmentation: The model showed significant gains in mapping static elements, particularly in detecting pedestrian crossings (+1.85 pts IoU) and lanes (+1.12 pts IoU).
  • Efficiency: While adding the image-space encoder increases the parameter count by about 5.3%, it provides a performance jump that justifies the slightly lower inference speed (1.94 FPS vs 2.18 FPS on an NVIDIA A6000).

Limitations and Future Work

  • Computational Overhead: The dual-aggregation approach is more computationally expensive than single-space methods. The authors note a ~11% decrease in FPS.
  • Camera-Specific Patterns: The authors experimented with "camera-specific" encoders (training different weights for front vs. side cameras) but found that the nuScenes dataset wasn't large enough to provide sufficient guidance for this, leading to mixed results.
  • Future Directions: The authors suggest that using more sophisticated, modern optical flow architectures or integrating these insights into even newer "foundation models" for driving could yield further improvements.

Summary updated

Abstract

Autonomous driving requires an accurate representation of the environment. A strategy toward high accuracy is to fuse data from several sensors. Learned Bird’s-Eye View (BEV) encoders can achieve this by mapping data from individual sensors into one joint latent space. For cost-efficient camera-only systems, this provides an effective mechanism to fuse data from multiple cameras with different views. Accuracy can further be improved by aggregating sensor information over time. This is especially important in monocular camera systems to account for the lack of explicit depth and velocity measurements, such that decision-critical information about distance distances and motions of other objects is easily accessible. Thereby, the effectiveness of developed BEV encoders crucially depends on the operators used to aggregate temporal information and on the used latent representation spaces.We analyze BEV encoders proposed in the literature and compare their effectiveness, quantifying the effects of aggregation operators and latent representations. While most existing approaches aggregate temporal information either in image or in BEV latent space, our analyses and performance comparisons suggest that these latent representations exhibit complementary strengths. Therefore, we develop a novel temporal BEV encoder, TempBEV, which integrates aggregated temporal information from both latent spaces. We consider subsequent image frames as stereo through time and leverage methods from optical flow estimation for temporal stereo encoding.Empirical evaluation on the NuScenes dataset shows a significant improvement by TempBEV over the baseline for 3D object detection and BEV segmentation. The ablation uncovers a strong synergy of joint temporal aggregation in the image and BEV latent space. These results indicate the overall effectiveness of our approach and make a strong case for aggregating temporal information in both image and BEV latent spaces.
BibTeX
@inproceedings{monninger2024tempbev,
  author = {Thomas Monninger and Vandana Dokkadi and Md Zafar Anwar and Steffen Staab},
  title = {{TempBEV: Improving Learned BEV Encoders with Combined Image and BEV Space Temporal Aggregation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801646},
}

Loading related content...

Figure