Lane detection using lane boundary marker network with road geometry constraints

Hussam Ullah Khan, Afsheen Rafaqat Ali, Ali Hassan, Ahmed Ali, Wajahat Kazmi, Aamer ZaheerWACV 2020

TL;DR

This paper introduces a novel lane detection system that combines a deep learning-based keypoint detector with geometric constraints to robustly identify lane boundaries, even in challenging scenarios like occlusions and missing markings. By leveraging the inherent parallelism and equidistance of lane lines, the method effectively predicts missing lanes and improves overall accuracy, outperforming state-of-the-art methods on several public datasets.

What Problem Does This Paper Address?

Accurate lane detection is a cornerstone for autonomous driving and advanced driver-assistance systems (ADAS). While deep learning has significantly advanced this field, current methods still struggle with scenarios where lane markings are occluded, faded, or entirely absent. These "no-line" or "shadowed" conditions pose a critical challenge, as they can lead to missed lane detections, potentially compromising safety. Existing approaches often rely on detecting lane boundaries directly or segmenting the entire lane region, but these can be brittle in the face of real-world complexities. This paper aims to address these limitations by developing a more robust lane detection system that can reliably infer lane positions even with limited visual evidence.

Methods

The proposed approach consists of two main phases: initialization and post-initialization.

Initialization: * Lane Boundary Keypoint Detection: A fully convolutional neural network (CNN) is employed to detect keypoints (markers) along the lane boundaries. This network uses an encoder-decoder architecture, with ResNet50 as the backbone for feature extraction. The keypoint detection is trained using a per-mask cross-entropy loss. * Camera Initialization: This phase estimates the Inverse Perspective Mapping (IPM) using road geometry. It involves: * Horizon Estimation: Fitting lines to detected markers and identifying vanishing points to estimate the horizon line. * IPM and Image Rectification: Computing a rectification homography to transform the image into a bird's-eye view, effectively removing perspective distortion. * Lane Width Initialization: Estimating the average lane width in pixels based on detected lane boundaries in the rectified view.

Post-Initialization: * Equidistant Parallel Line/Curve Fitting: The detected keypoints are rectified using the estimated IPM. Second-degree quadratic curves (or straight lines if curves fail) are fitted to these points, with constraints for parallelism and equidistance. This step ensures that the fitted lanes adhere to expected road geometry. * Detecting Weak or Missing Lane Boundaries: This is a core contribution. The system uses the initialized lane width to predict missing lane boundaries by calculating an offset from detected lanes. A reciprocal weighted averaging technique is then applied to combine predictions and detected lanes, giving more weight to lines with stronger evidence. This allows the system to infer lanes that were not initially detected by the keypoint network.

Datasets

The paper evaluates its approach on three publicly available datasets:

  • CULane [33]: A large dataset with 55 hours of traffic video, divided into eight scene categories.
  • Caltech [3]: An older dataset with four video sets, noted for inconsistent ground truth.
  • TuSimple [36]: A dataset with 3626 training and 2782 test images, known for its challenging scenarios.

The authors also mention that the TuSimple ground truth sometimes marks occluded lane pixels as background, which can lead to a "discontinued ground truth." They propose a modified evaluation on TuSimple (Ours40-gt Ref) to account for this.

Key Insights and Ideas

  • Geometric Constraints as a Powerful Prior: The paper strongly emphasizes that leveraging the inherent geometric properties of lanes (parallelism, equidistance) is crucial for robust lane detection, especially when visual evidence is scarce.
  • Keypoint Detection + Geometric Post-processing: Instead of relying solely on end-to-end deep learning for final lane prediction, the authors combine a deep keypoint detector with a geometric post-processing pipeline. This modularity allows for targeted improvements and better handling of edge cases.
  • Predicting the Unseen: The method's ability to infer missing lane boundaries by using initialized lane width and parallelism is a significant advancement, directly addressing a major limitation of many existing systems.
  • Weighted Averaging for Robustness: The reciprocal weighted averaging technique effectively balances confidence in detected lanes with predictions for missing ones, leading to more stable and accurate results.

Key Results

  • CULane: The proposed method (Ours50) achieved a significant improvement in F1 scores, outperforming state-of-the-art methods like SCNN and JLBNet. For instance, it showed improvements of +7.8% and +6.8% over the state-of-the-art on CU-Lane and Caltech datasets, respectively. Notably, it achieved an overall F1 score of approximately 81% on CULane, a substantial increase from previous benchmarks.
  • Caltech: The algorithm outperformed SCNN on the Caltech dataset, achieving F1 scores of 89.90% (15:15 mapping) and 96.8% (30:30 mapping).
  • TuSimple: While the standard evaluation showed a slight decrease in accuracy compared to SCNN (95.5% vs. 96.5%), a modified evaluation (Ours40-gt Ref) that accounts for ground truth discontinuities reached almost 98% accuracy, surpassing SCNN and SAD-Nets.
  • Qualitative Results: Figure 1 and Figure 6 demonstrate the algorithm's effectiveness in various challenging scenarios, including "No Line" classes, crowded conditions, and occluded lanes, showing recovered lane boundaries in red.

Limitations and Future Work

  • Limitations:
    • The current approach relies on the assumption of parallel and equidistant lane boundaries, which may not hold for auxiliary or loading lanes.
  • The performance on curved lane boundaries is noted as slightly lower than some state-of-the-art methods in certain categories.
  • The efficiency (FPS) is primarily for reference and was not the primary optimization target.
  • Future Work:
    • Address auxiliary and loading lanes where the parallelism assumption doesn't apply.
  • Improve the detection of curved lane boundaries.
  • Further optimize the system for efficiency.
  • Explore combining deep learning and geometric computer vision for a more comprehensive system.

Summary updated

BibTeX
@inproceedings{khan2020lane,
  author = {Hussam Ullah Khan and Afsheen Rafaqat Ali and Ali Hassan and Ahmed Ali and Wajahat Kazmi and Aamer Zaheer},
  title = {{Lane detection using lane boundary marker network with road geometry constraints}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure