Abstract
Bird's eye view (BEV) representation is a new perception formulation for autonomous driving, which is based on spatial fusion. Further, temporal fusion is also introduced in BEV representation and gains great success. In this work, we propose a new method that unifies both spatial and temporal fusion and merges them into a unified mathematical formulation. The unified fusion could not only provide a new perspective on BEV fusion but also brings new capabilities. With the proposed unified spatial-temporal fusion, our method could support long-range fusion, which is hard to achieve in conventional BEV methods. Moreover, the BEV fusion in our work is temporal-adaptive and the weights of temporal fusion are learnable. In contrast, conventional methods mainly use fixed and equal weights for temporal fusion. Besides, the proposed unified fusion could avoid information lost in conventional BEV fusion methods and make full use of features. Extensive experiments and ablation studies on the NuScenes dataset show the effectiveness of the proposed method and our method gains the state-of-the-art performance in the map segmentation task.
Introduction
In autonomous driving, Bird’s-Eye-View (BEV) perception has emerged as the gold standard for environmental understanding. Constructing an accurate BEV map requires fusing data across both space (multiple camera angles) and time (historical frames). However, existing methods predominantly rely on warp-based temporal fusion, which aligns past BEV features with the current vehicle position using ego-motion.
This traditional approach suffers from three critical limitations:
- Information Loss: Repeated warping and resampling of features leads to a "blurring" effect and loss of fine-grained detail.
- Short-Term Memory: Serial processing often causes long-range history to be forgotten or become excessively noisy.
- Rigid Weighting: Fixed or equal weights for past frames prevent the model from adaptively prioritizing the most relevant temporal information.
To address these, we introduce UniFusion, a Transformer-based framework that unifies spatial and temporal fusion into a single, parallel mathematical formulation.
Methodology: Unified Fusion via Virtual Views
The core innovation of UniFusion is the concept of Virtual Views. Instead of warping processed BEV features, we treat past camera images as additional "virtual" cameras currently existing in the environment. By leveraging the vehicle's known motion, we calculate extrinsic parameters for these past cameras relative to the current BEV coordinate system.
Key Components
- Unified Cross-Attention: By placing past and present views in the same coordinate system, a Transformer can use BEV queries to attend to all cameras across all time steps simultaneously. This enables parallel rather than serial fusion.
- Adaptive Fusion: The attention mechanism naturally learns to weight frames (past or present) based on their relevance to specific map regions.
- Self-Regression and "Grafting": The model concatenates previous Transformer outputs with current queries. This "grafts" successive Transformers together, effectively doubling the network's depth to improve temporal consistency without significant computational overhead.
Training Strategies
To manage the memory costs of long-range fusion, we employ Disentangled Training. We found that a model trained on short sequences (e.g., 2 past steps) can effectively generalize to much longer sequences (e.g., 6-10 steps) during inference.
Experimental Setup
We evaluated UniFusion on the NuScenes dataset using a more rigorous benchmark setting:
- Extended Range: We expanded the evaluation area to 160m × 100m to test the limits of long-range perception.
- City-based Split: To test real-world generalization, we proposed splitting data by city (e.g., training on Boston, validating on Singapore) rather than standard random splits, which often contain overlapping intersections.
Results and Discussion
UniFusion achieves state-of-the-art (SOTA) results on the NuScenes map segmentation task:
- Performance: On the standard 100m x 100m split, it achieved a Road mIoU of 85.4, significantly outperforming BEVFormer (80.1).
- Efficiency: UniFusion runs at 2.6-2.7 FPS on an RTX 3090, proving more efficient than many competitors (e.g., BEVFormer at 1.7 FPS).
- Temporal Robustness: Unlike warp-based methods that degrade after 3-4 frames, UniFusion’s accuracy continues to improve with up to 10 frames (2 seconds) of history.
Conclusion
UniFusion demonstrates that treating temporal fusion as a spatial problem via virtual views avoids the pitfalls of feature warping. While the model shows impressive performance and efficiency, challenges remain in generalization across different cities. Future work will explore heterogeneous fusion, integrating vehicle-side data with fixed infrastructure cameras to further enhance perception range and reliability.
Summary updated