Efficient Lane Boundary Detection with Spatial-Temporal Knowledge Filtering

Zhixiong Nan, Ping Wei, Linhai Xu, Nanning Zheng

TL;DR

This paper introduces a novel lane boundary detection method that leverages both visual appearance and prior spatial-temporal knowledge of road structures. By employing two new filters, the Crossing Point Filter (CPF) and the Structure Triangle Filter (STF), the method effectively removes noisy line segments, leading to robust lane detection even in challenging scenarios like poor lighting, worn markings, and complex road layouts.

What Problem Does This Paper Address?

Detecting lane boundaries in real-world driving scenarios is a critical yet challenging task for autonomous vehicles and advanced driver-assistance systems. Existing methods often struggle with dynamic changes in lane structures (e.g., widening or narrowing lanes), occlusions from other vehicles or shadows, and variations in visual cues (e.g., worn-out markings or strong glare). These issues can lead to unreliable lane detection, making conventional approaches insufficient. The core problem this paper addresses is the need for a more robust and accurate lane detection system that can overcome these limitations by incorporating contextual knowledge beyond just local image features.

Methods

The proposed method integrates feature extraction with a novel spatial-temporal knowledge filtering framework.

  • Feature Extraction: The paper utilizes the Line Segment Detector (LSD) [1] to extract line segments from video frames. LSD is chosen for its efficiency and ability to detect line segments without requiring manual parameter tuning. To enhance lane contrast, the RGB image is converted to grayscale with a specific weighting of red and green channels.
  • Vanishing Line Detection: A method is proposed to detect the vanishing line without camera calibration. This involves calculating crossing points of line segments, dividing the image into horizontal bands, scoring these bands based on crossing point density, and identifying the vanishing line as the symmetry axis of the highest-scoring band. Line segments above this vanishing line are then eliminated as noise.
  • Crossing Point Filter (CPF): This filter leverages the geometric property that parallel lines in 3D world converge at a vanishing point in 2D images. It defines a "vanishing box" near the vanishing point and filters out line segments whose crossing points with other segments fall outside this box. This helps remove segments that do not align with the expected lane convergence.
  • Structure Triangle Filter (STF): This filter addresses line segments that are parallel to lane boundaries but are not actual lane markings (e.g., markings on traffic signs). It assumes a consistent lane width and uses the concept of a "structure triangle" formed by neighboring lanes and the image bottom. It estimates key points (B, C, D, E) based on line segment slopes and positions, and filters out segments that fall outside a defined "tolerance region" around these estimated lane boundaries.
  • Temporal Knowledge Transition: To handle potential errors in STF estimation, the filter incorporates temporal consistency. If the estimated lane width in the current frame deviates significantly from a prior average, the STF from the previous frame is used, reflecting the temporal knowledge of lane structure.
  • Lane Fitting: A road state machine determines whether the lane is straight or curved. Based on this state, either a straight line fitting model (using slope and point estimation) or a curved lane fitting model (Catmull-Rom curve) is applied to the filtered line segments to produce the final lane boundaries. For curved lanes, a similarity measure between consecutive frames is used to select the most consistent curve.

Datasets

  • XJTU-IAIR Traffic Scene Dataset: The authors collected a large-scale, realistic traffic scene dataset comprising approximately 103,176 video frames. This dataset covers a wide range of challenging conditions, including various lane types (dashed, curve, worn-out), road structures (wide, narrow, merging, tunnels), disturbances (shadows, paint, vehicles, water), illumination variations (day, night, dazzling), and vehicle maneuvers (lane changes, turns).
  • Aly's Dataset [33]: This is a well-known dataset used for evaluating lane detection algorithms. The paper uses it for comparison with other methods.

Key Insights and Ideas

  • Prior Knowledge is Crucial: The paper strongly advocates that relying solely on appearance features is insufficient for robust lane detection. Incorporating spatial-temporal knowledge about road geometry and lane consistency significantly improves performance.
  • Two-Stage Filtering for Robustness: The combination of CPF (geometric convergence) and STF (structural consistency) provides a powerful two-stage filtering mechanism to eliminate diverse types of noise that single filters might miss.
  • Vanishing Point as a Geometric Anchor: The use of the vanishing point, even without precise calibration, serves as a critical geometric anchor for filtering line segments that are unlikely to be part of actual lane boundaries.
  • Temporal Consistency for Stability: The temporal knowledge transition in the STF helps maintain lane detection stability across frames, especially when visual cues are weak or temporarily misleading.
  • Adaptive Fitting: The road state machine allows the system to adapt its fitting strategy (straight vs. curved) based on the road geometry, leading to more accurate boundary representation.

Key Results

  • High Precision and Recall: The proposed method achieved excellent performance on the XJTU-IAIR dataset, with a precision of 99.9% on highways and high precision (95.4% to 97.7%) in other challenging urban and illumination conditions. Recall was also consistently high (98.4% to 100%).
  • Robustness to Challenging Scenarios: The method demonstrated strong performance in various difficult scenarios, including complex illumination, night driving, worn lane markings, and changing road structures, as illustrated in Figures 11 and 12.
  • Competitive Performance: When compared to other state-of-the-art methods (He's, Bertozzi's, Seo's) on the XJTU-IAIR dataset, the proposed method generally outperformed them, especially in challenging conditions, highlighting the benefit of its knowledge-based filtering.
  • Real-time Capability: The algorithm was successfully applied to an autonomous experimental vehicle and achieved FPS rates of around 18 on a Raspberry Pi 3 and 29 on an ARK-10 after optimization, indicating its potential for real-time applications.

Limitations and Future Work

  • Low Resolution Video: The paper acknowledges that low resolution of video frames can be a limitation, potentially leading to more false negatives in situations with very weak visual cues.
  • Special Traffic Conditions: The method may produce false results in specific scenarios like crossroads, zebra crossings, and wet roads, as shown in Figure 14.
  • Future Work: The authors plan to focus on addressing these limitations and expanding their research to other intelligent vehicle-related topics, including pedestrian action prediction, complex traffic scene understanding, and 3D traffic scene reconstruction.

Summary updated

BibTeX
@inproceedings{nan2016efficient,
  author = {Zhixiong Nan and Ping Wei and Linhai Xu and Nanning Zheng},
  title = {{Efficient Lane Boundary Detection with Spatial-Temporal Knowledge Filtering}},
  booktitle = {Sensors},
  year = {2016},
  url = {https://openreview.net/forum?id=TOPp9EpqJ6},
}
Figure