Introduction
Deep neural networks excel at semantic segmentation, but safety-critical applications like autonomous driving require models to know when they are likely incorrect. This paper introduces SynthCP (Synthesize then Compare), a unified framework designed to address two primary challenges: Failure Detection (identifying incorrect predictions) and Anomaly (OOD) Detection (identifying objects the model was never trained to recognize).
The SynthCP Framework
Unlike traditional methods that rely on pixel-level probability scores, SynthCP leverages the global structure of a scene. The core idea is a "reverse sanity check": if a segmentation mask is accurate, a generative model should be able to reconstruct the original image from that mask. Discrepancies between the original and reconstructed images signal potential failures or anomalies.
Methodology
The framework operates in two main stages:
- Image Synthesis Module: Using a conditional GAN (specifically the SPADE architecture), the system takes a predicted segmentation mask $\hat{y}$ and synthesizes a corresponding image $\hat{x}$.
- Comparison Module: The original image $x$ and synthesized image $\hat{x}$ are compared. For failure detection, a Siamese network predicts image-level IoU and pixel-level error maps. For anomaly detection, the system calculates the cosine distance between intermediate feature representations of both images.
Experimental Validation
The researchers evaluated SynthCP across three distinct domains: urban street scenes for failure detection (Cityscapes), pancreatic tumor datasets for segmentation quality prediction (Medical Segmentation Decathlon), and a benchmark for anomaly segmentation featuring 250 types of unseen objects (StreetHazards).
Key Results
SynthCP demonstrated significant improvements over state-of-the-art methods:
- Failure Detection: Improved pixel-level error prediction on Cityscapes by 6% (AUPR-Error).
- Medical Imaging: Achieved a 7% improvement in Pearson correlation for quality prediction when combined with shape-based models.
- Anomaly Segmentation: Set a new state-of-the-art on StreetHazards with a 20% improvement in AUPR.
Insights and Limitations
Key Insights:
- Synthesis as a Proxy for Confidence: The ability to reconstruct an image serves as a powerful indicator of model certainty. If a model claims a region is a "car" but the GAN synthesizes something else, the model is likely wrong.
- Structural Awareness: By using a GAN, the system captures structural relationships that pixel-wise probability maps often miss.
- Complementary Modalities: In medical contexts, SynthCP's focus on texture complements shape-based VAE methods.
Limitations:
- GAN Artifacts: Imperfect synthesis can lead to false alarms, where the GAN fails to reconstruct a region correctly despite an accurate segmentation.
- Computational Overhead: Running high-quality cGANs like SPADE is computationally intensive, posing challenges for real-time deployment in its current form.
Future work aims to improve synthesis fidelity and develop more sophisticated discrepancy networks to further reduce false positive rates.
Summary updated