FusionLane: Multi-Sensor Fusion for Lane Marking Semantic Segmentation Using Deep Neural Networks

Ruochen Yin, Yong Cheng, Huapeng Wu, Yuntao Song, Biao Yu, Runxin NiuIEEE Trans. ITS 2022 · DOI

TL;DR

This paper introduces FusionLane, a novel deep learning architecture that fuses LiDAR and camera data for highly accurate lane marking semantic segmentation. By leveraging a bird's-eye view representation from LiDAR and incorporating temporal information with ConvLSTM, FusionLane significantly improves the precision of lane marking detection, crucial for high-definition map construction in autonomous driving.

What Problem Does This Paper Address?

Building high-definition (HD) maps for autonomous driving is a critical but challenging task. Lane markings are a fundamental component of these maps, providing essential information for vehicle localization and navigation. While image-based semantic segmentation methods have advanced, they struggle to provide the precise 3D spatial positioning of lane markings due to the inherent limitations of cameras. LiDAR, on the other hand, offers accurate depth information but can be noisy and less effective at capturing fine details. Existing methods often rely solely on one sensor, failing to meet the stringent accuracy requirements for HD map construction. This paper aims to bridge this gap by proposing a multi-sensor fusion approach that combines the strengths of both LiDAR and cameras.

Methods

The core of FusionLane is a deep neural network that fuses information from two distinct sensor modalities:

  • Camera Data (C-Region): Front-view camera images are first transformed into a bird's-eye view (CBEV) using inverse perspective mapping. This CBEV is then semantically segmented using a pre-trained DeepLabV3+ model. This segmentation provides classification information about lane markings.
  • LiDAR Data (LBEV): LiDAR point clouds are also converted into a bird's-eye view representation (LBEV). Unlike simple projection, this LBEV is a three-channel image encoding intensity, average height, and height standard deviation of the LiDAR points within each grid cell. This representation captures precise spatial information.

The FusionLane network employs a dual-input encoder where the processed C-Region and LBEV data are fed into separate branches. These branches are fused at multiple stages, allowing the network to learn to combine classification cues from the camera with spatial information from LiDAR.

A key innovation is the integration of a ConvLSTM module within the network. This module leverages the temporal continuity of driving scenes by processing sequential feature maps. This helps to mitigate errors and blind spots in the C-Region segmentation by using historical context.

Finally, a decoder module upsamples the fused features to reconstruct the final semantic segmentation map, incorporating low-level features from the LiDAR branch to enhance detail recovery.

Datasets

The authors constructed their own dataset based on the KITTI dataset. This dataset includes synchronized LiDAR and camera data. They manually labeled 436 images of both LBEV and CBEV. For training and testing, they expanded this to 14,720 labeled LBEV and CBEV pairs by rotating images. The dataset includes six labels for CBEV (Background, Solid Line, Dotted Line, Stop Line, Arrow, Prohibited Area) and an additional "Other Point" category for LBEV.

Key Insights and Ideas

  • LiDAR Bird's-Eye View (LBEV) as the primary segmentation target: Shifting the segmentation target from camera images to LBEV is crucial for obtaining accurate 3D spatial positions, a key requirement for HD maps.
  • Dual-branch encoder for independent sensor processing: Using separate input branches for LiDAR and camera data allows the network to learn features independently before fusion, accommodating potential misalignments between sensor modalities.
  • ConvLSTM for temporal context: Incorporating temporal information via ConvLSTM is vital for improving robustness, especially in handling errors or occlusions in the camera-based segmentation.
  • FusionLane outperforms single-sensor approaches: The experiments clearly demonstrate that combining LiDAR and camera data significantly outperforms methods relying on either sensor alone.

Key Results

  • Superior performance on LBEV segmentation: The proposed FusionLane model achieved the highest Mean Intersection over Union (MIOU) of 85.35% on the testing set, outperforming other models by a significant margin.
  • Improved accuracy across categories: FusionLane achieved the best IOU for most lane marking categories, including Solid Line, Dotted Line, Arrow, and Stop Line, with an overall MIOU of 85.35%.
  • Effective error mitigation: The ConvLSTM module demonstrably reduced errors compared to a model without temporal information (FusionLane_Without_LSTM).
  • Visual validation: Qualitative results showed FusionLane producing nearly perfect semantic segmentation maps, effectively recovering details and correctly classifying various lane marking types.

Limitations and Future Work

  • Dataset size: While the authors created a substantial dataset, they acknowledge the need for more training data.
  • Future work: The authors plan to:
  • Transform the "C-Region" input from CBEV to a Bird's-Eye View (BEV) representation for improved consistency with the LBEV input.
  • Focus on acquiring more training data.
  • Begin constructing high-precision maps using their developed method.

Summary updated

BibTeX
@article{yin2022fusionlane,
  author = {Ruochen Yin and Yong Cheng and Huapeng Wu and Yuntao Song and Biao Yu and Runxin Niu},
  title = {{FusionLane: Multi-Sensor Fusion for Lane Marking Semantic Segmentation Using Deep Neural Networks}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2022},
  doi = {10.1109/tits.2020.3030767},
}
Figure