Epipolar Attention Field Transformers for Bird's Eye View Semantic Segmentation

Christian Witte, Jens Behley, Cyrill Stachniss, Marvin Raaijmakers

Spatial understanding is the bedrock of autonomous driving. To make safe decisions, a self-driving car needs to transform raw data from multiple cameras into a unified, top-down "Bird's Eye View" (BEV) of its surroundings. While Transformer-based models have become the gold standard for this task, they often rely on "black box" learned positional encodings to figure out how image pixels relate to the BEV grid.

A new paper, "Epipolar Attention Field Transformers for Bird’s Eye View Semantic Segmentation," argues that we shouldn't force models to learn geometry from scratch. Instead, we should bake the laws of physics—specifically epipolar geometry—directly into the architecture.

TL;DR

EAFormer replaces brittle, learned positional encodings in BEV transformers with explicit geometric priors called Epipolar Attention Fields (EAFs). By weighting attention based on known camera parameters, the model achieves state-of-the-art accuracy and demonstrates vastly superior generalization when transferred to entirely new camera setups.

What Problem Does This Paper Address?

Most modern vision-only BEV systems use Transformers to map 2D image features to a 3D-aware BEV grid. To help the model understand where a pixel in a camera image belongs on the ground plane, researchers typically use positional encodings. These are learned vectors added to the data that implicitly represent the camera's configuration.

The problem? This approach is data-hungry and fragile. If you change the camera's mounting angle or move the model to a different vehicle with a different sensor suite, the learned encodings often fail. The model has "memorized" one specific geometry rather than understanding the underlying spatial relationship.

Methods

The authors introduce EAFormer, which ditches learned positional encodings in favor of Epipolar Attention Fields (EAFs).

The core idea relies on a classic computer vision principle: epipolar geometry. If you have a point on the BEV grid, that point must project onto a specific line (the epipolar line) in the camera's view.

Figure 2. The BEV (grey) is considered the reference view for epipolar geometry. We project each coordinate of the grid cell cen- ter onto all other views. A given grid cell (red) thus corresponds to the epipolar lines projected onto the source views 1 and 2. We im- pose a normal distribution on the distance to the epipolar line and obtain Epipolar Attention Fields, which are visualized as heatmaps in the source views.

However, because the BEV grid is discretized (coarse), a single line isn't enough. The authors propose "fields"—Gaussian distributions imposed on the distance to the epipolar line. These fields are used to weight the Transformer's cross-attention mechanism:

  1. Explicit Weighting: Instead of adding a vector, the attention scores are multiplied by the EAF weights. High weights are given to image features near the epipolar line, and low weights to those far away.
  2. Scale Awareness: The width of the Gaussian field is adjusted based on the distance from the camera, accounting for the fact that closer grid cells project to wider areas in the image.
Figure 4. An illustration of our EAFormer architecture. For each input image, we obtain multi-scale image feature maps. We compute the Epipolar Attention Fields (EAF) based on the image feature coordinates, the BEV feature grid coordinates, and the camera parameters. The Epipolar Transformer Encoder attends the BEV grid queries to the image feature iteratively over the multi-scale resolutions and weights the output by the EAF. A semantic head (decoder) upsamples the output of this attention-based encoding and predicts the final output.

Datasets

The model was trained and evaluated on two major autonomous driving benchmarks:

  • nuScenes: A large-scale dataset with 1,000 scenes captured in Boston and Singapore using a 6-camera setup.
  • Argoverse 2 (AV2): Another massive dataset featuring 1,000 sequences from six US cities, utilizing a 7-camera ring setup.

Crucially, the authors performed cross-dataset evaluation, training on one and testing on the other to measure how well the geometric priors generalize to unseen camera configurations.

Key Insights and Ideas

  • Geometry as a Strong Prior: By using EAFs, the model doesn't have to "guess" where to look in an image for a specific BEV cell; the math tells it exactly where the relevant features must lie.
  • Robustness to Change: Because the EAFs are calculated using camera intrinsics and extrinsics at runtime, the model can adapt to new camera setups (like a different car model) without retraining the entire network.
  • Discretization Matters: Moving from an infinitely thin epipolar line to a Gaussian "field" is essential for handling the coarse nature of feature maps and BEV grids, as well as minor calibration errors or vehicle pitch (e.g., driving uphill).

Key Results

EAFormer shows significant improvements over baseline methods like Cross-View Transformers (CVT):

  • Higher Accuracy: On the nuScenes validation set, EAFormer achieved 78.0% mIoU for drivable area segmentation, outperforming CVT by nearly 4%.
  • Superior Generalization: In zero-shot transfer tests (training on AV2 and testing on nuScenes), EAFormer's performance was four times higher than CVT's. While CVT's learned encodings failed to translate to the new camera rig, EAFormer's geometric logic held firm.
Qualitative comparison of zero-shot transfer. EAFormer maintains a coherent vehicle segmentation even when moved to a new dataset, while CVT's predictions degrade significantly.

Limitations and Future Work

  • Single Timestep: The current version of EAFormer processes images frame-by-frame. It does not yet incorporate temporal information from previous frames, which is vital for handling occlusions and predicting motion.
  • Future Directions: The authors suggest that temporal fusion could also be improved by imposing similar geometric constraints across time, potentially leading to even more robust 4D (3D + time) environmental understanding.

For more details, you can find the full paper under the title "Epipolar Attention Field Transformers for Bird’s Eye View Semantic Segmentation" (arXiv:2412.01595).

Summary updated

Abstract

Spatial understanding of the semantics of the surroundings is a key capability needed by autonomous cars to enable safe driving decisions. Recently, purely vision-based solutions have gained increasing research interest. In particular, approaches extracting a bird's eye view (BEV) from multiple cameras have demonstrated great performance for spatial understanding. This paper addresses the dependency on learned positional encodings to correlate image and BEV feature map elements for transformer-based methods. We propose leveraging epipolar geometric constraints to model the relationship between cameras and the BEV by Epipolar Attention Fields. They are incorporated into the attention mechanism as a novel attribution term, serving as an alternative to learned positional encodings. Experiments show that our method EAFormer outperforms previous BEV approaches by 2% mIoU for map semantic segmentation and exhibits superior generalization capabilities compared to implicitly learning the camera configuration.
BibTeX
@inproceedings{witte2025epipolar,
  author = {Christian Witte and Jens Behley and Cyrill Stachniss and Marvin Raaijmakers},
  title = {{Epipolar Attention Field Transformers for Bird's Eye View Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure