RETR: A Structure-Preserving RGB-Event Transformer for Robust 3D Lane Detection

Jingtao Dong, Hao Zhuang, Hao Yang, Liyuan Pan, Wei LiangarXiv 2026 · arXiv

TL;DR

RETR is the first framework to bring event cameras into the world of 3D lane detection, overcoming the limitations of traditional RGB cameras in "blind" conditions like glare or motion blur. By using a structure-preserving transformer to fuse high-dynamic-range event data with standard images, the authors achieved state-of-the-art accuracy on two new multimodal benchmarks, particularly in challenging lighting and complex road geometries.

What Problem Does This Paper Address?

Reliable 3D lane detection—knowing exactly where lane markings are in terms of lateral position, distance, and elevation—is a "must-have" for autonomous driving. However, current systems rely almost exclusively on RGB cameras.

In the real world, RGB cameras are fragile. Lane markings are thin and low-contrast; they disappear in the glare of a tunnel exit, get washed out by headlights at night, or turn into a smeary mess due to motion blur at high speeds. While researchers have tried to fix this with better geometric reasoning, the underlying visual evidence is often just too weak.

Enter event cameras. Unlike standard cameras that capture full frames at fixed intervals, event cameras only record changes in brightness at the pixel level with microsecond latency. They have an incredibly high dynamic range (HDR), meaning they can "see" structure in shadows or bright light where an RGB sensor would just see black or white. The challenge? Event data is sparse and motion-dependent, making it notoriously difficult to fuse with standard images into a coherent 3D map.

Methods

The authors propose RETR (a structure-preserving RGB-Event TRansformer), which moves away from simple feature concatenation toward a "reliability-to-geometry" pipeline. The architecture consists of three main novel components:

  1. Reciprocal Context Flow Fusion (RCF): Instead of just mashing RGB and event features together, RCF uses a cross-attention mechanism. It identifies "modality-consistent" evidence—areas where both sensors agree—and allows them to exchange context bidirectionally.
  2. Uncertainty Aware Structural Consolidation (UASC): Lane markings are thin. Standard neural network operations (like pooling) often "smear" these fine details. UASC uses an uncertainty gate—driven by entropy and high-frequency cues—to decide which spatial regions need extra structural refinement, ensuring thin and faint lane evidence isn't lost during feature aggregation.
  3. Geometry State Decoder (GSD): This component treats lanes as ordered geometric entities. It initializes "lane queries" based on proposals and evolves them using predicted 3D coordinates. This allows the model to refine the lane's shape in 3D space iteratively rather than just classifying pixels.

Datasets

Because no 3D lane detection datasets with event data existed, the authors built two new benchmarks:

  • DSEC-3DLD: A real-world dataset built on the DSEC driving sequences. It features synchronized RGB, event, and LiDAR data, which the authors painstakingly converted into metric 3D lane annotations.
  • Ev-OpenLane: A large-scale simulated benchmark. The authors used the ESIM event simulator on the popular OpenLane dataset to generate 200,000 frames of synthetic event data, providing a scalable way to train and test the model.

Key Insights and Ideas

  • Events as Structural Anchors: The "aha moment" here is that event cameras don't just provide more data; they provide different data. In a dark tunnel, the RGB sensor might fail, but the event camera captures the high-contrast edges of the lane markings perfectly.
  • Fighting the "Runge Phenomenon": The authors noted that fitting 3D curves directly to image-plane data often causes "oscillatory artifacts" (wiggles) at long ranges. Their solution was to lift 2D points into 3D space first and then perform refinement in the metric ground-coordinate system, leading to much smoother and more accurate distant predictions.
  • Uncertainty-Guided Refinement: By explicitly modeling where the model is "unsure" (using entropy), RETR can selectively apply more computational power to reconstruct the most difficult parts of the road.

Key Results

RETR significantly outperformed existing RGB-only methods across the board:

  • Overall Performance: On the real-world DSEC-3DLD benchmark, RETR achieved an F1 score of 75.6%, a 6.4-point jump over the previous best RGB-only baseline (LATR).
  • Robustness to Lighting: The gains were most dramatic in "edge cases." In overexposed scenes, the F1 score improved by 10.6 points, and in underexposed scenes, it jumped by 11.4 points.
  • Geometric Accuracy: The model reduced the centerline distance error to 0.094m, demonstrating that the fusion of events helps not just with detection, but with precise localization.

Limitations and Future Work

While a major step forward, the authors acknowledge a few hurdles:

  • Data Scarcity: Real-world event data with high-quality 3D labels is still rare. The field needs more diverse, large-scale real datasets to move beyond simulation.
  • Computational Overhead: Fusing two modalities naturally increases the parameter count and latency. While the authors profiled RETR on an RTX 4090, further optimization is needed for low-power embedded systems found in most cars.
  • Sensor Configuration: The current work focuses on a specific RGB-event setup. Future research could explore how different sensor placements or resolutions affect the "structure-preserving" capabilities of the transformer.

Reference: Dong, J., et al. "RETR: A Structure-Preserving RGB-Event Transformer for Robust 3D Lane Detection." 40th Conference on Neural Information Processing Systems (NeurIPS 2026).

Summary updated

Abstract

Robust 3D lane detection requires accurate metric road geometry recovery from visual inputs, yet conventional RGB-based approaches suffer from fragile lane evidence under adverse illumination, motion blur, and long-range perspective compression. Event cameras offer complementary high-temporal-resolution, high dynamic-range structural cues that are robust to these challenges, but their sparse, motion-dependent responses cannot be directly fused into consistent 3D geometry. In this paper, we present the first attempt to introduce event cameras into 3D lane detection. To enable systematic research, we first build two multimodal benchmarks with metric 3D lane annotations: DSEC-3DLD (real-world sequences) and Ev-OpenLane (large-scale simulated sequences). We further propose RETR, a structure preserving transformer that converts complementary RGB and event observations into coherent 3D lane geometry. RETR first aligns modality consistent evidence through Reciprocal Context Flow Fusion, then preserves thin and uncertain lane structures with Uncertainty Aware Structural Consolidation, and finally decodes ordered lane hypotheses using a Geometry State Decoder with proposal conditioned initialization, reference conditioned query evolution, and diversified geometric inquiry. Extensive experiments show that RETR achieves state-of-the-art performance on both benchmarks, especially showing strong improvements in challenging lighting and complex road geometry scenarios. Code and dataset will be available at https://github.com/duskswell/RETR.
BibTeX
@misc{dong2026retr,
  author = {Jingtao Dong and Hao Zhuang and Hao Yang and Liyuan Pan and Wei Liang},
  title = {{RETR: A Structure-Preserving RGB-Event Transformer for Robust 3D Lane Detection}},
  year = {2026},
  eprint = {2611.19358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19358},
}
Figure