CARLANE: A Lane Detection Benchmark for Unsupervised Domain Adaptation from Simulation to multiple Real-World Domains

Bonifaz Stuhr, Johann Haselberger, Julian GebeleNeurIPS 2022

Lane detection is a cornerstone of autonomous driving, but it suffers from a severe "data bottleneck." Labeling thousands of real-world images with precise lane coordinates is incredibly expensive and time-consuming. While developers can generate infinite labeled data in simulators like CARLA, models trained purely on "perfect" synthetic data often fail when they hit the messy, unpredictable real world. This phenomenon is known as domain shift.

Unsupervised Domain Adaptation (UDA) aims to fix this by transferring knowledge from a labeled source (simulation) to an unlabeled target (the real world). However, while UDA has flourished in object detection and semantic segmentation, it has been largely neglected in lane detection due to a lack of standardized, public benchmarks. The CARLANE benchmark was designed specifically to fill this void, providing a comprehensive pipeline to evaluate how well models can bridge the "sim-to-real" gap.

The CARLANE Benchmark and Datasets

To rigorously test UDA methods, the researchers introduced three distinct datasets totaling 163K images. These datasets challenge models to adapt from CARLA simulations to both 1/8th scale model cars and full-scale highway driving.

Examples of the MoLane and TuLane datasets showing the domain shift from simulation (top) to real-world targets (bottom).
  1. MoLane (Simulation → 1/8th Model Car): Focuses on abstract lane markings in a controlled but real physical environment using a small-scale vehicle.
  2. TuLane (Simulation → TuSimple): Targets full-scale U.S. highway driving, using the well-known TuSimple dataset as the target domain.
  3. MuLane (Multi-Target): A highly challenging "3-way" setup where a single model must adapt from simulation to both the model car and the highway domains simultaneously.

To help others expand the dataset, the authors also released a custom labeling tool and a waypoint-based agent for CARLA.

Methodology: Evaluating UDA Strategies

The authors established a rigorous evaluation framework using Ultra Fast Structure-aware Deep Lane Detection (UFLD) as their baseline architecture. They evaluated several established UDA strategies against this baseline:

  • Adversarial Methods (DANN, ADDA): These utilize a "discriminator" network to force the features from the simulation and the real world to become indistinguishable to the main neural network.
  • Self-Training (SGADA): This approach uses the model’s own high-confidence predictions on real-world data as "pseudo-labels" to iteratively refine itself.
  • SGPCS (The Authors' Method): The authors propose Self-supervised Prototypical Cross-domain Self-supervised Learning, which combines prototypical clustering (grouping similar features together) with pseudo-labeling to achieve state-of-the-art results on the benchmark.

Key Insights and Results

The quantitative results demonstrate that while UDA significantly improves performance over source-only training, there is still a long way to go to match fully supervised models.

Lane detection attention maps comparing source and target domains across MoLane, TuLane, and MuLane.
  • The "UDA Gap": Even the best UDA methods still lag behind fully supervised models by a significant margin. On MoLane, the authors' SGPCS method achieved 93.94% accuracy (a solid jump from the 89.39% "Source-Only" baseline). However, a fully supervised model hit ~97.3%, confirming that lane detection UDA is far from a solved problem.
  • The Multi-Target Wall: Adapting to one real-world domain is hard; adapting to two at once is significantly harder. In the MuLane (multi-target) dataset, SGPCS reached 91.57% accuracy, but False Positive rates remained high (around 45%). This indicates that current UDA methods struggle with "domain interference" and often "hallucinate" lanes in complex real-world scenes.
  • The Danger of Negative Transfer: In some tests, the popular DANN method actually performed worse than a model trained only on simulation data. This "negative transfer" happens when the domain shift is too complex, causing the model to learn features that are actively detrimental to the target task.

Limitations and Future Work

The authors acknowledge a few constraints in their current benchmark. The 1/8th scale tracks used a fixed set of elements and lacked complex features like buildings or traffic signs. Furthermore, the synthetic data was limited to existing CARLA maps rather than custom-designed environments tailored for specific edge cases.

Ultimately, the primary takeaway from CARLANE is the urgent need for more robust UDA methods. Future research must focus on improving multi-target adaptation and finding ways to reduce the "hallucination" of lanes in unfamiliar real-world environments, ensuring the high precision required for safe autonomous driving.

Summary updated

BibTeX
@inproceedings{stuhr2022carlane,
  author = {Bonifaz Stuhr and Johann Haselberger and Julian Gebele},
  title = {{CARLANE: A Lane Detection Benchmark for Unsupervised Domain Adaptation from Simulation to multiple Real-World Domains}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure