Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Gläser, Fabian Timm, Werner Wiesbeck, Klaus Dietmayer

TL;DR

This paper provides a comprehensive survey of deep multi-modal object detection and semantic segmentation for autonomous driving. It systematically reviews existing datasets, fusion methodologies, and highlights key challenges and open questions in the field, offering a valuable resource for researchers and practitioners.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate and robust perception of their surroundings. This perception is typically achieved by fusing data from multiple sensors (like cameras, LiDAR, and radar) to overcome the limitations of individual sensors. However, the field of deep multi-modal perception for autonomous driving lacks a unified understanding of how to best combine these sensor modalities. Specifically, questions remain about what data to fuse, when to fuse it within a neural network architecture, and how to perform the fusion effectively. This paper aims to bridge this gap by providing a structured overview and analysis of the current state-of-the-art, datasets, methods, and challenges.

Methods

The paper categorizes fusion methodologies into three main stages:

  • Early Fusion: Raw or pre-processed sensor data is fused at the input layer. This allows the network to learn joint features early on but is inflexible to changes in sensor input and sensitive to misalignment.
  • Late Fusion: Decisions from individual sensor-specific networks are combined at the output. This offers high flexibility and modularity but can be computationally expensive and discards rich intermediate features.
  • Middle Fusion: Features are fused at intermediate layers of the neural network. This offers a compromise between early and late fusion, allowing for learning cross-modal features at various levels of abstraction. The paper further distinguishes between "deep fusion" (hierarchical fusion across multiple layers) and "short-cut fusion" (direct fusion of features from different modalities at specific layers).

The paper also discusses various fusion operations like addition, average mean, concatenation, ensemble, and Mixture of Experts (MoE), highlighting that concatenation is the most common operation.

Datasets

The paper reviews numerous multi-modal datasets for autonomous driving research. Key datasets mentioned include:

  • KITTI [6], [75]: A widely used benchmark for various autonomous driving tasks, including object detection and semantic segmentation.
  • nuScenes [89]: A large-scale dataset featuring 3D LiDAR, cameras, and radar, with comprehensive annotations.
  • Oxford RobotCar Dataset [74], [85]: Collected over a year, it offers diverse weather and lighting conditions.
  • Astyx HiRes2019 Datasets [94]: Includes radar data.
  • KAIST Multispectral Dataset [93]: Provides both thermal and visual camera data.

The paper notes that while many datasets exist, they often suffer from limitations such as limited driving scenarios, sensor variety, and imbalanced object classes. The use of virtual datasets generated from game engines (e.g., GTA-V) is also discussed as a way to overcome data limitations.

Key Insights and Ideas

  • Multi-modality is Crucial: The paper strongly emphasizes that fusing data from multiple sensors is essential for achieving robust and accurate perception in autonomous driving, outperforming single-sensor approaches.
  • Fusion Stage Matters: The choice of when to fuse (early, middle, or late) significantly impacts performance, flexibility, and computational cost. Middle fusion appears to offer a good balance.
  • Data Quality and Alignment are Paramount: The accuracy and robustness of multi-modal perception systems are highly dependent on the quality of data labeling and precise spatial and temporal alignment of sensor data.
  • Radar is Under-explored: While cameras and LiDAR are commonly fused, the use of radar data in deep multi-modal perception is still nascent, presenting a significant area for future research.
  • Uncertainty Modeling is Key for Robustness: Explicitly modeling sensor uncertainty is crucial for safe autonomous driving, especially in adverse conditions or when sensors degrade. This is an active area of research.

Key Results

The paper does not present new experimental results but rather synthesizes findings from existing literature. It highlights that:

  • Methods fusing camera and LiDAR information generally achieve better performance than those using either sensor alone.
  • Two-stage object detectors (like Faster R-CNN) tend to offer higher accuracy but at the cost of inference time compared to one-stage detectors (like YOLO, SSD).
  • The performance of different fusion methods is highly dependent on the specific modalities, data, and network architectures used.

Limitations and Future Work

  • Data Diversity and Quality: The paper identifies a need for more diverse multi-modal datasets covering a wider range of scenarios, weather conditions, and sensor combinations. Improving data labeling efficiency and robustness to noisy labels are also critical.
  • Fusion Methodologies: There's a lack of clear guidelines on optimal fusion strategies ("what," "when," and "how" to fuse). Research is needed to compare different feature representations and fusion operations, especially for under-utilized sensors like radar and ultrasonics.
  • Robustness and Uncertainty: Developing methods to explicitly model and propagate sensor uncertainties is crucial for improving robustness against sensor degradation, adverse conditions, and open-set scenarios.
  • Real-time Performance: Achieving real-time perception while maintaining accuracy and robustness remains a significant challenge, requiring efficient network architectures and fusion operations.
  • Evaluation Metrics: Current metrics primarily focus on accuracy and do not adequately capture robustness or performance in challenging conditions. New metrics are needed.
  • Network Architectures: Exploring more advanced architectures, including those that incorporate temporal cues (e.g., RNNs) and end-to-end learning approaches, is a promising direction.

Summary updated

Abstract

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of "what to fuse", "when to fuse", and "how to fuse" remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.
BibTeX
@article{feng2021deep,
  author = {Di Feng and Christian Haase-Schütz and Lars Rosenbaum and Heinz Hertlein and Claudius Gläser and Fabian Timm and Werner Wiesbeck and Klaus Dietmayer},
  title = {{Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges}},
  journal = {IEEE Trans. ITS},
  year = {2021},
  doi = {10.1109/tits.2020.2972974},
}
Figure