TL;DR
Lidar sensors often "see" noise instead of objects during rain or snow, leading to dangerous perception failures in autonomous vehicles. This paper demonstrates that by processing sequences of lidar frames and using a novel "temporal offset" augmentation (randomly skipping frames during training), deep learning models can significantly improve their detection accuracy in adverse weather without requiring complex manual filtering.
What Problem Does This Paper Address?
Lidar is a cornerstone of autonomous driving, providing precise 3D maps of the environment. However, its reliability plummets in adverse weather. Raindrops, fog, and snowflakes cause laser beams to scatter or reflect prematurely, creating "noisy" point clouds full of artifacts and missing data.
Most current solutions try to fix this by adding pre-processing filters (to "de-noise" the data) or by fusing lidar with other sensors like radar. This paper asks a different question: Can we make the lidar detector itself more robust by looking at how the world changes over time? Since weather noise is often transient and random, while physical objects move in predictable ways, temporal sequences should—in theory—help a model distinguish a car from a flurry of snow.
Methods
The researchers took a state-of-the-art Pillar-based Object Detection (PBOD) model as their baseline and extended it to handle sequences of lidar frames. They tested several architectural strategies:
- Input Concatenation (IC): Merging raw points from multiple time steps into one giant point cloud.
- Feature Concatenation (FC): Extracting features from each frame separately and then merging them later in the neural network.
- Recurrent Neural Networks (LSTM): Using "Convolutional Long Short-Term Memory" (convLSTM) cells to allow the network to maintain a "memory" of previous frames.
The most significant contribution is a new augmentation strategy called Temporal Offset. During training, the model doesn't just look at consecutive frames (e.g., frame 1, 2, 3). Instead, it randomly skips frames (e.g., frame 1, 4, 7). This forces the model to learn more generalized motion patterns and prevents it from overfitting to the specific frame rate of a single dataset.
Datasets
The authors evaluated their methods on three real-world datasets specifically chosen for their challenging weather conditions:
- DENSE: Captured in Germany, Denmark, Sweden, and Finland; features rain, fog, and snow.
- Canadian Adverse Driving Conditions (CADC): A specialized dataset focused entirely on snowy driving in Waterloo, Canada.
- nuScenes: A massive, popular dataset containing various weather conditions (mostly rain and clear) from Boston and Singapore.
Key Insights and Ideas
- Raw Concatenation is a Trap: Simply "stacking" raw point clouds (Input Concatenation) actually performed worse than the baseline. Without accounting for the ego-vehicle's motion or abstracting features first, the overlapping points created a "ghosting" effect that confused the detector.
- The Power of Randomness: The "Temporal Offset" augmentation was the MVP. By varying the time gap between frames, the model became much better at generalizing. It stopped looking for specific pixel-level changes and started understanding the underlying physics of moving objects.
- LSTMs for Snow: While Feature Concatenation worked well for rain and fog, the convLSTM approach was particularly effective for the CADC (snowy) dataset. This suggests that "memory" is crucial when objects are frequently obscured by heavy, drifting snow.
Key Results
The proposed methods outperformed the baseline across all three datasets:
- nuScenes: The modified model with convLSTM and temporal offset saw a massive 22.2% increase in mean Average Precision (mAP) at an IoU of 0.5.
- CADC (Snow): The LSTM-based model improved mAP by 6.1%, showing better resilience to heavy snowfall.
- DENSE: Feature Concatenation with temporal offset improved performance by 4.1%.
However, these gains come with a "speed tax." The inference time increased from roughly 0.1 seconds per iteration to about 0.16–0.2 seconds, which might require further optimization for real-time deployment on vehicle hardware.
Limitations and Future Work
The authors identified two main areas for improvement:
- Coordinate Alignment: The current "early-stage" concatenation methods don't account for the movement of the sensor between frames. Adding coordinate transformations could fix the "ghosting" issues observed in the IC models.
- Architecture Evolution: While LSTMs were effective, the authors suggest that Transformer architectures (which are currently dominating other areas of AI) might be even better at capturing long-range temporal dependencies in lidar data.
- Efficiency: Future work needs to bridge the gap between the increased accuracy of temporal models and the strict real-time latency requirements of autonomous vehicles.
Summary updated