CurveFormer++: 3D Lane Detection by Curve Propagation with Temporal Curve Queries and Attention

Yifeng Bai, Zhirong Chen, Pengpeng Liang, Bo Song, Erkang ChengarXiv 2024 · arXiv

TL;DR

CurveFormer++ is a single-stage, Transformer-based 3D lane detection method that eliminates the need for complex and error-prone perspective-to-Bird's-Eye-View (BEV) view transformations. By modeling lanes as sparse curve queries with dynamic 3D anchor points and introducing a lane-centric temporal fusion mechanism, it achieves state-of-the-art accuracy and temporal stability on major benchmarks.

What Problem Does This Paper Address?

Accurate 3D lane detection is a foundational requirement for autonomous driving tasks like path planning, lane-keeping assist, and navigation. Traditional methods typically follow a two-stage pipeline: they first project front-facing camera images into a dense Bird's-Eye-View (BEV) space (using Inverse Projective Mapping or depth estimation) and then apply a convolutional network or Transformer to detect the lanes.

However, this dense view transformation is a major bottleneck. It heavily relies on accurate, real-time camera extrinsics and flat-ground assumptions. When a vehicle encounters uphill or downhill sections, these assumptions break down, leading to severe geometric distortions and error propagation. Furthermore, because lanes are inherently sparse structures, computing a dense BEV map wastes massive amounts of computational power on empty space.

Comparison of different 3D lane detection pipelines.

Methods

CurveFormer++ bypasses the BEV transformation entirely, directly regressing 3D lane parameters from perspective image features. The architecture consists of three core innovations:

  1. Sparse Curve Queries with Dynamic 3D Anchor Points: Instead of dense queries, lanes are represented as sparse curve queries associated with an ordered set of 3D anchor points. These points are iteratively refined layer-by-layer within the Transformer decoder, allowing the model to gradually converge on the true 3D geometry of the lane.
  2. Context Sampling & Range Restriction: Standard Deformable-DETR models predict feature sampling offsets purely from the query. CurveFormer++ introduces a Context Sampling Module that combines both the query and local image features to calculate more accurate sampling offsets. Additionally, an Anchor Point Range Restriction module dynamically predicts the start and end positions of lanes, preventing the model from processing irrelevant image features for shorter lanes.
  3. Lane-Centric Temporal Fusion (CurveFormer++-T): To leverage historical frame data without dense BEV maps, the authors introduce a temporal propagation block. It takes the Top-K highest-confidence curve queries and anchor points from the previous frame, transforms them using ego-motion data, and injects them into the current frame's decoder layers via temporal self-attention.
Fig. 3. Overview of our proposed CurveFormer++ for single-frame 3D lane detection (a) & temporal propagation fusion block in CurveFormer++-T (b).

Datasets

The authors evaluated CurveFormer++ on two prominent real-world 3D lane detection benchmarks:

  • OpenLane Dataset: A massive dataset containing over 200K frames (157K training, 39K validation) based on the Waymo Open Dataset, complete with camera intrinsics and extrinsics.
  • ONCE-3DLanes Dataset: A dataset containing 211K images featuring high-quality 3D lane annotations across diverse weather conditions and geographic regions.

Key Insights and Ideas

  • Bypassing BEV Restores Vertical Awareness: By avoiding the flat-ground assumption inherent in traditional image-to-BEV projections, CurveFormer++ retains vertical information, making it significantly more robust in uphill, downhill, and undulating road scenarios.
  • Sparse Temporal Fusion is More Efficient: Aligning dense BEV feature maps across time is difficult and noisy because lanes occupy only a fraction of the grid. Propagating sparse, high-confidence object queries frame-by-frame provides a much cleaner and computationally efficient temporal signal.
  • Decoder-Only Streamlining: The authors discovered that replacing the traditional heavy encoder-decoder structure with a decoder-only network reduced computational overhead by 13.8% while incurring a negligible performance drop (0.1% F1-score).

Key Results

  • ONCE-3DLanes Benchmark: CurveFormer++ achieved a 77.22% F1-score (single-frame) and 77.85% (with temporal fusion), outperforming the popular CNN-based Anchor3DLane (74.44%) and the Transformer-based PersFormer (72.07%).
  • OpenLane Benchmark: The model achieved a 52.7% F1-score, beating PersFormer by 2.2% across the entire validation set, showing particularly strong performance gains in complex scenarios like night driving and sharp curves.
  • Temporal Stability: The temporal variant (CurveFormer++-T) dramatically reduced prediction jitter. It demonstrated a lower standard deviation in X-axis prediction error compared to PersFormer across 96% of evaluated video segments.

Limitations and Future Work

Despite its strong performance, CurveFormer++ relies entirely on a single front-view monocular camera. This creates inherent limitations when dealing with long-distance detections, sharp bends, or highly uneven road surfaces where visual features become severely degraded or occluded.

To address this, the authors point to two promising future directions: extending the architecture to support multi-camera inputs for full 360-degree contextual awareness, and incorporating multi-modal data (such as 3D LiDAR geometry) to further anchor the model's spatial understanding in highly challenging environments.

Summary updated

Abstract

In autonomous driving, accurate 3D lane detection using monocular cameras is important for downstream tasks. Recent CNN and Transformer approaches usually apply a two-stage model design. The first stage transforms the image feature from a front image into a bird's-eye-view (BEV) representation. Subsequently, a sub-network processes the BEV feature to generate the 3D detection results. However, these approaches heavily rely on a challenging image feature transformation module from a perspective view to a BEV representation. In our work, we present CurveFormer++, a single-stage Transformer-based method that does not require the view transform module and directly infers 3D lane results from the perspective image features. Specifically, our approach models the 3D lane detection task as a curve propagation problem, where each lane is represented by a curve query with a dynamic and ordered anchor point set. By employing a Transformer decoder, the model can iteratively refine the 3D lane results. A curve cross-attention module is introduced to calculate similarities between image features and curve queries. To handle varying lane lengths, we employ context sampling and anchor point restriction techniques to compute more relevant image features. Furthermore, we apply a temporal fusion module that incorporates selected informative sparse curve queries and their corresponding anchor point sets to leverage historical information. In the experiments, we evaluate our approach on two publicly real-world datasets. The results demonstrate that our method provides outstanding performance compared with both CNN and Transformer based methods. We also conduct ablation studies to analyze the impact of each component.
BibTeX
@misc{bai2024curveformer,
  author = {Yifeng Bai and Zhirong Chen and Pengpeng Liang and Bo Song and Erkang Cheng},
  title = {{CurveFormer++: 3D Lane Detection by Curve Propagation with Temporal Curve Queries and Attention}},
  year = {2024},
  eprint = {2402.06423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.06423},
}

Loading related content...

Figure