Lane and Road Marker Semantic Video Segmentation Using Mask Cropping and Optical Flow Estimation

Guansheng Xing, Ziming ZhuSensors 2021 · DOI

TL;DR

This paper introduces a novel deep learning approach for semantic video segmentation of lanes and road markers. By leveraging temporal information through mask cropping and optical flow, the method significantly improves segmentation stability and accuracy in video sequences, outperforming single-frame methods and offering a good balance between performance and speed.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurately perceiving the environment, with lane and road marker segmentation being a critical component. While single-frame segmentation methods have shown promise, they often produce unstable results when applied to video sequences due to their inability to capture temporal continuity. Existing multi-frame methods can suffer from error accumulation or are too computationally expensive for real-time applications. This paper aims to bridge this gap by developing a robust and efficient method that effectively utilizes information from adjacent frames to achieve stable and accurate lane and road marker segmentation in videos.

Methods

The proposed method employs a multi-input, single-output, end-to-end trainable network that integrates information from consecutive frames. The core of the approach involves several key steps:

  • Optical Flow Estimation: An off-the-shelf optical flow estimation network (PWC-Net) is used to calculate the motion between the current and previous frames. This provides crucial information about how pixels have moved.
  • Mask Cropping: The segmentation result from the previous frame is used to generate a mask. This mask, after being dilated based on the estimated optical flow, is applied to the current frame to "crop" an area of interest. This focuses the network's attention on regions where lanes and road markers are likely to appear.
  • Main Network: This network takes both the cropped image and the full current frame as inputs. It processes these through separate branches (Clipping-net branch for the cropped image and a standard CNN branch for the full image) to extract features. These features are then fused to produce a preliminary segmentation prediction for the current frame.
  • Warping and Optimization Network: The segmentation result from the previous frame is warped to the current frame's position using the calculated optical flow. This warped segmentation, along with the preliminary prediction from the main network, is fed into an "optimized network." This network refines the segmentation, leveraging temporal consistency to correct errors and produce a more stable output.

The paper emphasizes that no post-processing is used, aiming for real-time performance without frame delays.

Datasets

The proposed method was evaluated on two publicly available datasets:

  • Baidu Apolloscape Lanemark Segmentation Dataset: This dataset is described as being collected from multiple cities in China and contains tens of thousands of labeled images with a high resolution (3384 × 2710). It includes various lane markings, turning arrows, zebra crossings, and parking area boundaries, with precise annotations derived from 3D point clouds.
  • CULane Dataset: This dataset is a large-scale, challenging dataset for lane detection, collected from over 55 hours of video footage. It features diverse scenarios, including varying lighting conditions (overpasses, night scenes), crowded traffic, and obscured lane markings.

Key Insights and Ideas

  • Leveraging Temporal Context is Crucial for Stability: The paper strongly advocates that relying solely on single frames leads to unstable segmentation in video. By explicitly incorporating information from previous frames, the model can achieve significantly smoother and more reliable results.
  • Mask Cropping as a "Region Proposal" Mechanism: The innovative use of the previous frame's segmentation to crop the current frame acts as an intelligent way to guide the network. It highlights areas of high probability for targets, reducing computational load and focusing attention on relevant regions.
  • Optical Flow as a Temporal Alignment Tool: Optical flow is not just used for motion estimation but is actively employed to warp previous frame information to the current frame. This temporal alignment is key to the optimization step, allowing for effective fusion of past and present information.
  • Two-Stage Training for Better Convergence: The authors found that training the main network first to converge before introducing the optimized network significantly improved learning efficiency and reduced noise in the final output. This suggests that a stable initial prediction is essential for the refinement stage.
  • Targeted Improvement for Difficult Scenarios: The method shows particular strength in challenging scenarios like dazzle light, arrows, and night scenes, where single-frame methods struggle. This highlights the value of temporal context for occluded or low-visibility targets.

Key Results

The proposed method demonstrated significant improvements across several metrics:

  • Speed: The segmentation speed of video lanes and road markers was accelerated by 2.5 times compared to previous methods.
  • Accuracy:
    • mIoU classes: Increased by 1.4% compared to the best single-frame method (DeepLabv3+).
  • mIoU categories: Increased by 1.5% compared to the best single-frame method.
  • On the Apolloscape dataset, the method achieved 0.776 mIoU cls, 0.953 mIoU cat, and 0.771 TC.
  • On the CULane dataset, the method achieved 0.783 mIoU and 0.783 TC, outperforming several state-of-the-art methods.
  • Temporal Consistency (TC): Improved by 6.5% compared to the best single-frame method. In the worst case, temporal consistency was reduced by only 2.2% at most, indicating a good trade-off.
  • Qualitative Improvements: The method produced smoother results, suppressed segmentation instability, and more accurately segmented challenging targets like turning arrows, broken lines, and zebra crossings, especially in difficult lighting or occluded scenarios.

Limitations and Future Work

  • Limitations: The authors acknowledge that for "normal or other scenes" where lanes are clearly visible and inter-frame movement has little effect, their method does not significantly improve accuracy. This is because the method's primary strength lies in addressing occluded or difficult-to-see lanes and temporal instability.
  • Future Work: The authors plan to continue studying the segmentation and detection of lanes and road markers. Their goal is to explore novel pipelines that leverage continuous frame information to further enhance the segmentation accuracy and stability of target edges in image sequences.

Summary updated

Abstract

Lane and road marker segmentation is crucial in autonomous driving, and many related methods have been proposed in this field. However, most of them are based on single-frame prediction, which causes unstable results between frames. Some semantic multi-frame segmentation methods produce error accumulation and are not fast enough. Therefore, we propose a deep learning algorithm that takes into account the continuity information of adjacent image frames, including image sequence processing and an end-to-end trainable multi-input single-output network to jointly process the segmentation of lanes and road markers. In order to emphasize the location of the target with high probability in the adjacent frames and to refine the segmentation result of the current frame, we explicitly consider the time consistency between frames, expand the segmentation region of the previous frame, and use the optical flow of the adjacent frames to reverse the past prediction, then use it as an additional input of the network in training and reasoning, thereby improving the network's attention to the target area of the past frame. We segmented lanes and road markers on the Baidu Apolloscape lanemark segmentation dataset and CULane dataset, and present benchmarks for different networks. The experimental results show that this method accelerates the segmentation speed of video lanes and road markers by 2.5 times, increases accuracy by 1.4%, and reduces temporal consistency by only 2.2% at most.
BibTeX
@inproceedings{xing2021lane,
  author = {Guansheng Xing and Ziming Zhu},
  title = {{Lane and Road Marker Semantic Video Segmentation Using Mask Cropping and Optical Flow Estimation}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21217156},
}
Figure