Enhancing Lidar-Based Object Detection in Adverse Weather Using Offset Sequences in Time

Raphael van Kempen, Tim Rehbronn, Abin Jose, Johannes Stegmaier, Bastian Lampe, Timo Woopen, Lutz Eckstein

TL;DR

Lidar sensors often "see" noise instead of objects during rain or snow, leading to dangerous perception failures in autonomous vehicles. This paper demonstrates that by processing sequences of lidar frames and using a novel "temporal offset" augmentation (randomly skipping frames during training), deep learning models can significantly improve their detection accuracy in adverse weather without requiring complex manual filtering.

What Problem Does This Paper Address?

Lidar is a cornerstone of autonomous driving, providing precise 3D maps of the environment. However, its reliability plummets in adverse weather. Raindrops, fog, and snowflakes cause laser beams to scatter or reflect prematurely, creating "noisy" point clouds full of artifacts and missing data.

Most current solutions try to fix this by adding pre-processing filters (to "de-noise" the data) or by fusing lidar with other sensors like radar. This paper asks a different question: Can we make the lidar detector itself more robust by looking at how the world changes over time? Since weather noise is often transient and random, while physical objects move in predictable ways, temporal sequences should—in theory—help a model distinguish a car from a flurry of snow.

Methods

The researchers took a state-of-the-art Pillar-based Object Detection (PBOD) model as their baseline and extended it to handle sequences of lidar frames. They tested several architectural strategies:

  • Input Concatenation (IC): Merging raw points from multiple time steps into one giant point cloud.
  • Feature Concatenation (FC): Extracting features from each frame separately and then merging them later in the neural network.
  • Recurrent Neural Networks (LSTM): Using "Convolutional Long Short-Term Memory" (convLSTM) cells to allow the network to maintain a "memory" of previous frames.

The most significant contribution is a new augmentation strategy called Temporal Offset. During training, the model doesn't just look at consecutive frames (e.g., frame 1, 2, 3). Instead, it randomly skips frames (e.g., frame 1, 4, 7). This forces the model to learn more generalized motion patterns and prevents it from overfitting to the specific frame rate of a single dataset.

Datasets

The authors evaluated their methods on three real-world datasets specifically chosen for their challenging weather conditions:

  1. DENSE: Captured in Germany, Denmark, Sweden, and Finland; features rain, fog, and snow.
  2. Canadian Adverse Driving Conditions (CADC): A specialized dataset focused entirely on snowy driving in Waterloo, Canada.
  3. nuScenes: A massive, popular dataset containing various weather conditions (mostly rain and clear) from Boston and Singapore.

Key Insights and Ideas

  • Raw Concatenation is a Trap: Simply "stacking" raw point clouds (Input Concatenation) actually performed worse than the baseline. Without accounting for the ego-vehicle's motion or abstracting features first, the overlapping points created a "ghosting" effect that confused the detector.
  • The Power of Randomness: The "Temporal Offset" augmentation was the MVP. By varying the time gap between frames, the model became much better at generalizing. It stopped looking for specific pixel-level changes and started understanding the underlying physics of moving objects.
  • LSTMs for Snow: While Feature Concatenation worked well for rain and fog, the convLSTM approach was particularly effective for the CADC (snowy) dataset. This suggests that "memory" is crucial when objects are frequently obscured by heavy, drifting snow.

Key Results

The proposed methods outperformed the baseline across all three datasets:

  • nuScenes: The modified model with convLSTM and temporal offset saw a massive 22.2% increase in mean Average Precision (mAP) at an IoU of 0.5.
  • CADC (Snow): The LSTM-based model improved mAP by 6.1%, showing better resilience to heavy snowfall.
  • DENSE: Feature Concatenation with temporal offset improved performance by 4.1%.

However, these gains come with a "speed tax." The inference time increased from roughly 0.1 seconds per iteration to about 0.16–0.2 seconds, which might require further optimization for real-time deployment on vehicle hardware.

Limitations and Future Work

The authors identified two main areas for improvement:

  1. Coordinate Alignment: The current "early-stage" concatenation methods don't account for the movement of the sensor between frames. Adding coordinate transformations could fix the "ghosting" issues observed in the IC models.
  2. Architecture Evolution: While LSTMs were effective, the authors suggest that Transformer architectures (which are currently dominating other areas of AI) might be even better at capturing long-range temporal dependencies in lidar data.
  3. Efficiency: Future work needs to bridge the gap between the increased accuracy of temporal models and the strict real-time latency requirements of autonomous vehicles.

Summary updated

Abstract

Automated vehicles require an accurate perception of their surroundings for safe and efficient driving. Lidar-based object detection is a widely used method for environment perception, but its performance is significantly affected by adverse weather conditions such as rain and fog. In this work, we investigate various strategies for enhancing the robustness of lidar-based object detection by processing sequential data samples generated by lidar sensors. Our approaches leverage temporal information to improve a lidar object detection model, without the need for additional filtering or pre-processing steps. We compare $10$ different neural network architectures that process point cloud sequences including a novel augmentation strategy introducing a temporal offset between frames of a sequence during training and evaluate the effectiveness of all strategies on lidar point clouds under adverse weather conditions through experiments. Our research provides a comprehensive study of effective methods for mitigating the effects of adverse weather on the reliability of lidar-based object detection using sequential data that are evaluated using public datasets such as nuScenes, Dense, and the Canadian Adverse Driving Conditions Dataset. Our findings demonstrate that our novel method, involving temporal offset augmentation through randomized frame skipping in sequences, enhances object detection accuracy compared to both the baseline model (Pillar-based Object Detection) and no augmentation.
BibTeX
@inproceedings{kempen2023enhancing,
  author = {Raphael van Kempen and Tim Rehbronn and Abin Jose and Johannes Stegmaier and Bastian Lampe and Timo Woopen and Lutz Eckstein},
  title = {{Enhancing Lidar-Based Object Detection in Adverse Weather Using Offset Sequences in Time}},
  booktitle = {ICECET},
  year = {2023},
  doi = {10.1109/icecet58911.2023.10389489},
}
Figure