TempBEV: Improving Learned BEV Encoders with Combined Image and BEV Space Temporal Aggregation

Thomas Monninger, Vandana Dokkadi, Md Zafar Anwar, Steffen Staab

Abstract

Autonomous driving requires an accurate representation of the environment. A strategy toward high accuracy is to fuse data from several sensors. Learned Bird's-Eye View (BEV) encoders can achieve this by mapping data from individual sensors into one joint latent space. For cost-efficient camera-only systems, this provides an effective mechanism to fuse data from multiple cameras with different views. Accuracy can further be improved by aggregating sensor information over time. This is especially important in monocular camera systems to account for the lack of explicit depth and velocity measurements. Thereby, the effectiveness of developed BEV encoders crucially depends on the operators used to aggregate temporal information and on the used latent representation spaces. We analyze BEV encoders proposed in the literature and compare their effectiveness, quantifying the effects of aggregation operators and latent representations. While most existing approaches aggregate temporal information either in image or in BEV latent space, our analyses and performance comparisons suggest that these latent representations exhibit complementary strengths. Therefore, we develop a novel temporal BEV encoder, TempBEV, which integrates aggregated temporal information from both latent spaces. We consider subsequent image frames as stereo through time and leverage methods from optical flow estimation for temporal stereo encoding. Empirical evaluation on the NuScenes dataset shows a significant improvement by TempBEV over the baseline for 3D object detection and BEV segmentation. The ablation uncovers a strong synergy of joint temporal aggregation in the image and BEV latent space. These results indicate the overall effectiveness of our approach and make a strong case for aggregating temporal information in both image and BEV latent spaces.

Introduction

In autonomous driving, monocular (camera-only) systems are cost-effective but struggle with 3D perception because a single 2D image lacks explicit depth and velocity information. To address this, researchers use temporal aggregation: combining information from multiple frames over time to "triangulate" depth (temporal stereo) and estimate object motion.

Conceptual comparison of temporal motion in Image Space versus Bird's-Eye View (BEV) Space.

Current state-of-the-art models typically perform this aggregation in one of two places. Image Space offers high resolution but is complicated by ego-motion, making correspondences difficult to track. Conversely, BEV Space provides a top-down view where ego-motion is easily canceled, but it often suffers from low resolution and "lifting" errors during the 2D-to-3D conversion. TempBEV identifies that these two spaces are complementary rather than redundant.

Methodology

TempBEV integrates temporal information from both latent spaces simultaneously to capture both precise short-term motion and long-term environmental consistency.

The TempBEV architecture featuring parallel image-space and BEV-space temporal aggregation.

Image Space Aggregation (Short-term)

The model employs a "temporal stereo encoder" based on the FlowNet architecture. By processing pairs of consecutive images, it extracts high-resolution features related to optical flow. This captures the sharp motion cues necessary for dynamic object detection that are often blurred when projecting directly to BEV.

BEV Space Aggregation (Long-term)

Building on the BEVFormer architecture, TempBEV utilizes a recurrent mechanism. It maintains a persistent "memory" of the environment in BEV space, using temporal self-attention to update the current state with previous time steps. This approach is ideal for maintaining static map elements and handling occlusions over extended periods.

Synergistic Fusion

The features from the image-space stereo encoder are "lifted" into 3D and projected into the BEV grid. These are then fused with the recurrent BEV features through parallel aggregation, ensuring the final representation benefits from both the high-resolution motion cues of the image space and the spatial stability of the BEV space.

Experimental Results

TempBEV was evaluated on the nuScenes dataset, a standard benchmark for autonomous driving. The model was compared against a strong BEVFormer baseline and various other temporal operators.

  • 3D Object Detection: TempBEV achieved 51.31% NDS and 41.26% mAP, representing an improvement of +1.06 pts and +1.44 pts respectively over the BEVFormer baseline.
  • BEV Segmentation: The model showed significant gains in mapping static elements, particularly for pedestrian crossings (+1.85 pts IoU) and lanes (+1.12 pts IoU).
  • Optical Flow Pre-training: A key finding was that pre-training the image-space encoder on the Flying Chairs optical flow dataset significantly boosted 3D detection performance, confirming that learning 2D motion is a powerful prerequisite for 3D understanding.

Conclusion

TempBEV demonstrates that joint temporal aggregation in both image and BEV latent spaces provides a superior representation for autonomous driving tasks. While the dual-aggregation approach introduces a slight computational overhead (~11% decrease in FPS), the resulting performance gains in both detection and segmentation justify the cost. Future work may involve integrating more modern optical flow architectures or scaling the approach within larger foundation models for autonomous navigation.

Summary updated

Abstract

Autonomous driving requires an accurate representation of the environment. A strategy toward high accuracy is to fuse data from several sensors. Learned Bird's-Eye View (BEV) encoders can achieve this by mapping data from individual sensors into one joint latent space. For cost-efficient camera-only systems, this provides an effective mechanism to fuse data from multiple cameras with different views. Accuracy can further be improved by aggregating sensor information over time. This is especially important in monocular camera systems to account for the lack of explicit depth and velocity measurements. Thereby, the effectiveness of developed BEV encoders crucially depends on the operators used to aggregate temporal information and on the used latent representation spaces. We analyze BEV encoders proposed in the literature and compare their effectiveness, quantifying the effects of aggregation operators and latent representations. While most existing approaches aggregate temporal information either in image or in BEV latent space, our analyses and performance comparisons suggest that these latent representations exhibit complementary strengths. Therefore, we develop a novel temporal BEV encoder, TempBEV, which integrates aggregated temporal information from both latent spaces. We consider subsequent image frames as stereo through time and leverage methods from optical flow estimation for temporal stereo encoding. Empirical evaluation on the NuScenes dataset shows a significant improvement by TempBEV over the baseline for 3D object detection and BEV segmentation. The ablation uncovers a strong synergy of joint temporal aggregation in the image and BEV latent space. These results indicate the overall effectiveness of our approach and make a strong case for aggregating temporal information in both image and BEV latent spaces.
BibTeX
@inproceedings{monninger2024tempbev,
  author = {Thomas Monninger and Vandana Dokkadi and Md Zafar Anwar and Steffen Staab},
  title = {{TempBEV: Improving Learned BEV Encoders with Combined Image and BEV Space Temporal Aggregation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801646},
}
Figure