TL;DR
This paper introduces a novel "label-free" method for detecting failures in LiDAR-based point cloud segmentation models by comparing outputs from supervised and self-supervised learning streams. This approach leverages abundant unlabeled data to identify model weaknesses in both regular and outlier scenarios, and introduces LidarCODA, a new dataset for evaluating such anomaly detection capabilities.
What Problem Does This Paper Address?
Autonomous vehicles generate vast amounts of data, but current machine learning models are typically trained and validated on relatively small, curated datasets. This limited evaluation scope means that many potential model failure modes, especially those occurring in rare or "corner case" scenarios, go undetected during development. While large amounts of unlabeled data are readily available, effectively using this data to identify model failures is a significant challenge. The paper aims to bridge this gap by developing a method to detect these failures without relying on extensive labeled anomaly datasets, which are expensive and difficult to create.
Methods
The core of the proposed method is "complementary learning." The authors train two distinct models for the same task – semantic motion segmentation of LiDAR point clouds – using different learning paradigms:
- Supervised Stream: A standard supervised semantic segmentation model is trained to classify points as static or dynamic, leveraging human-annotated labels.
- Self-Supervised Stream: A self-supervised approach is employed, utilizing scene flow estimation and odometry data to predict future point cloud motion. This stream learns from the temporal dynamics inherent in the data without explicit labels.
The key idea is that these two streams, trained on different data characteristics, will exhibit disagreements when one of them fails. By comparing the point-wise predictions of the supervised and self-supervised models, the system can identify points where their outputs contradict. These discrepancies are then clustered and analyzed to detect and classify potential model failures.
Datasets
- Training: KITTI-360 and sequences from KITTI Odometry were used for training the supervised models.
- Evaluation:
- KITTI Odometry: Used for qualitative evaluation of regular scenarios.
- LidarCODA: A newly introduced dataset specifically for evaluating anomaly detection in LiDAR data. It's based on the CODA dataset but provides point-wise 3D labels in LiDAR space, addressing a limitation of the original CODA dataset which only offered 2D bounding boxes. LidarCODA includes subsets derived from ONCE, KITTI, and nuScenes datasets.
Key Insights and Ideas
- Complementary Learning for Failure Detection: The fundamental insight is that by training models with different inductive biases (supervised vs. self-supervised), their failure modes will likely differ, creating detectable disagreements.
- Leveraging Unlabeled Data: The approach effectively utilizes the abundance of unlabeled data, which is readily available in real-world driving scenarios, to identify model weaknesses that would be missed by traditional evaluation methods.
- LidarCODA as a Novel Benchmark: The creation of LidarCODA is significant, providing a much-needed resource for quantitatively evaluating anomaly detection in LiDAR point clouds, a critical aspect for autonomous driving safety.
- Disagreements Highlight Specific Weaknesses: The analysis shows that model disagreements often pinpoint specific types of failures, such as issues with distinguishing between static and dynamic objects in challenging temporal scenarios or with novel/unrecognizable objects.
Key Results
- Regular Scenarios: The method successfully identifies model failures in regular driving scenarios, which are often missed by standard evaluation. A qualitative analysis of over 20,000 frames revealed specific recurring weaknesses in both the supervised and self-supervised streams.
- Outlier Scenarios: The approach demonstrates increased sensitivity to anomalies, as evidenced by higher rates of model disagreement on outlier-rich datasets like LidarCODA compared to regular datasets like KITTI Odometry.
- Quantitative Evaluation: Using metrics like mIoU, AP, AR, and F1 score on LidarCODA, the paper shows that the method is particularly effective at detecting anomalies related to cyclists and "misc" (unrecognizable) objects, while struggling slightly with the "animal" class due to its scarcity.
- Dataset Performance: The method shows strong performance on LidarCODA subsets derived from ONCE and KITTI but struggles with the nuScenes subset, likely due to significant domain shifts in sensor setup.
Limitations and Future Work
- Simultaneous Model Failures: The paper acknowledges that if both the supervised and self-supervised streams make the same error (i.e., are consistently wrong), the model failure will go undetected. This is a known limitation of disagreement-based methods.
- Clustering Dependency: The self-supervised stream relies on a clustering strategy, which can potentially lead to faulty clusters and misclassifications.
- Future Work: The authors suggest applying this complementary learning approach to other tasks beyond motion segmentation, such as full panoptic segmentation, lane detection, or drivable-area segmentation, by adapting the training paradigms. They also highlight the potential for using detected failures to collect targeted training data for improving model robustness.
Summary updated