TL;DR
This paper introduces DaCUP, a novel method for detecting anomalies (like debris or potholes) on roads for autonomous driving. It achieves state-of-the-art results by effectively modeling diverse road appearances and ensuring detected anomalies are truly "unpredictable" and not just variations of known road surfaces, significantly reducing false positives.
What Problem Does This Paper Address?
Autonomous driving systems need to reliably identify "anomalies" on the road – objects or conditions that deviate from what's expected and could pose a hazard. This is a challenging anomaly detection problem. Traditional approaches often struggle because:
- Limited Training Data: Anomalies are rare and diverse, making it hard to collect comprehensive training data.
- Modeling Complexity: Road surfaces themselves can have varied appearances (asphalt, gravel, wet, dry), making it difficult to distinguish genuine anomalies from normal, but diverse, road conditions.
- False Positives: Many methods generate a high number of false positives, flagging normal road features (like lane markings or shadows) as anomalies, which is unacceptable for safety-critical systems.
This paper aims to address these issues by proposing a method that focuses on the inherent properties of anomalies: they are not from a class that can be easily modeled (i.e., not part of the "normal" road appearance) and they are not similar to their surrounding non-anomalous regions.
Methods
DaCUP is built upon an auto-encoder architecture and introduces several key innovations:
- Embedding Bottleneck: Instead of a simple bottleneck, DaCUP uses an "explicit embedding bottleneck" that learns a multi-modal representation of road appearances. This allows the model to better capture the diversity of normal road surfaces, even when trained on multiple datasets.
- Image-Conditioned Distance Features: This novel feature helps identify anomalies on-the-fly by calculating a "distance-to-class score." It measures how much a given image patch's embedding deviates from the average embedding of known road pixels in the current image. This improves the ability to distinguish true anomalies from false positives.
- Inpainting Module: To further reduce false positives and ensure anomaly consistency, an inpainting module is used. The core idea is that a true anomaly should not be reconstructible from its neighboring pixels. The method uses a pre-trained inpainting network (DeepFillv2) to identify regions that are too dissimilar to their surroundings to be inpainted, thus flagging them as potential anomalies. This module can also be used as a post-processing step for other anomaly detection methods.
The overall architecture involves a fixed semantic segmentation network, the embedding bottleneck, a reconstruction module, the distance-based scoring feature, and an inpainting module, all integrated into a coupling module that outputs the final anomaly score map.
Datasets
The paper utilizes several publicly available datasets for training and evaluation:
- Training: Cityscapes, BDD100k, and combinations thereof.
- Evaluation:
- Road Anomaly Detection:
- LaFRAROFS (a meta-dataset combining Lost-and-Found, Road Anomaly, Road Obstacles, and Fishyscapes).
- Road Anomaly Detection:
- Obstacle Track+ (a meta-dataset including data from the SegmentMeIfYouCan benchmark).
- SMIYC (SegmentMeIfYouCan) benchmark.
- Maritime Scenario: MaSTr1325 (for training) and MODS (for evaluation).
Key Insights and Ideas
- Anomalies are "Unpredictable": The paper's core insight is that anomalies are not just different, but fundamentally unpredictable from the context of normal data. This leads to the idea of using inpainting to verify this unpredictability.
- Multi-modal Road Modeling is Crucial: Simply having a compact representation of "road" isn't enough. The model needs to understand the variety of road appearances to avoid misclassifying normal variations as anomalies.
- Inpainting as a General Anomaly Filter: The inpainting module's ability to identify regions that cannot be reconstructed from their neighbors is a powerful, general mechanism for reducing false positives, applicable beyond just road anomaly detection.
- Joint Training Benefits: While the inpainting module can be used as a post-processor, jointly training it with the main method yields even better results, suggesting a deeper integration is beneficial.
Key Results
- State-of-the-Art Performance: DaCUP achieves state-of-the-art results on several road anomaly detection benchmarks, including ranking among the top 3 on the SMIYC benchmark and top 1 on the LaFRAROFS dataset.
- Significant False Positive Reduction: The inpainting module, in particular, is shown to significantly reduce false positives (FPR95) across various methods, as demonstrated in Table 3.
- Improved Data Utilization: The explicit embedding bottleneck improves the utilization of diverse training data, leading to better modeling of multi-modal road appearances and lower false positives.
- Maritime Application: The method demonstrates competitive performance in a maritime scenario for obstacle detection, highlighting its domain-agnostic nature.
- Processing Speed: The paper reports processing times, with DaCUP achieving a reasonable speed of 0.80 seconds per image on an NVIDIA GeForce RTX 2080 Ti.
Limitations and Future Work
- Small, Distant Objects: The paper acknowledges that DaCUP's performance is weaker for detecting small, distant objects. This is hypothesized to be due to the low resolution of features in the embedding space and the lack of lateral skip connections during upsampling in certain modules.
- Inpainting Domain Mismatch: When applying the inpainting module to a new domain (like maritime), its performance can degrade if it wasn't trained on data from that domain, leading to hallucinations of false positives.
- Future Work: The authors suggest exploring architectures with higher resolution features and lateral skip connections to improve small object detection. Further research into domain-adaptive inpainting could also be beneficial.
Summary updated