Introduction
For Level 5 autonomous vehicles, reliable vision is a prerequisite for safety. While significant research focuses on identifying external objects like pedestrians and vehicles, a critical but often overlooked challenge is the degradation of the sensors themselves. Wide-angle fisheye cameras, typically mounted low on the vehicle for parking and low-speed maneuvers, are highly susceptible to environmental contaminants such as mud, rain, road spray, and frost.
When a lens becomes soiled, standard perception algorithms—including semantic segmentation and object detection—degrade rapidly. To mitigate this, vehicles require robust soiling detection algorithms to trigger hardware cleaning systems. However, developing these algorithms faces a significant data bottleneck: soiling is a relatively rare event in standard datasets, and manually annotating blurry, semi-transparent contaminants is expensive and subjective. This paper introduces DirtyGAN, a generative framework designed to synthesize realistic soiling patterns and provide pixel-perfect ground-truth labels automatically.
Methodology
The researchers propose two primary architectural approaches to bridge the data gap for lens soiling:
- Baseline CycleGAN Pipeline: This approach utilizes a CycleGAN to learn the domain transformation from "clean" to "soiled." To avoid common GAN artifacts that alter the entire image, the authors use a separate segmentation network to isolate the generated "dirt" and a super-resolution network for scaling. The synthetic dirt is then blended onto high-resolution clean images using Gaussian-blurred masks to simulate realistic soft edges.
- DirtyGAN (VAE-Integrated): To achieve finer control over soiling types, the authors integrated a Variational AutoEncoder (VAE). By encoding real-world soiling patterns into a lower-dimensional latent space, the model can "walk" along the manifold to generate an infinite variety of unique, realistic patterns that maintain the visual logic of real grime.
The Power of Automatic Annotation
The most transformative aspect of the DirtyGAN framework is its ability to generate automatic annotations. Because the system synthesizes the dirt, it possesses exact knowledge of which pixels are affected. This eliminates the "human factor" in labeling messy, ambiguous boundaries. Furthermore, the VAE's latent space allows for interpolation between different patterns, potentially enabling the generation of synthetic video data where mud splashes dry or raindrops move with temporal consistency.
Results and Evaluation
The impact of GAN-based augmentation on model performance is substantial. Training a detection model on a combination of real and synthetic data improved the Mean Intersection over Union (mIoU) from 73.95% to 91.71%—an absolute increase of nearly 18%.
Key findings from the experimental phase include:
- Perception Degradation: Standard semantic segmentation performance drops by approximately 22% when a lens is soiled, confirming the necessity of detection systems.
- The Turing Test for Grime: In subjective tests, human experts could not distinguish between real and synthetic soiling 95% of the time.
- Generalization: Although trained on fisheye images from the WoodScape dataset, the GAN performed effectively on pinhole camera images from the Cityscapes dataset, suggesting that the visual physics of lens soiling are largely independent of camera intrinsics.
Conclusion and Future Work
While DirtyGAN significantly advances the state of the art for opaque soiling like mud, challenges remain. Semi-transparent contaminants, such as water droplets, involve complex optical refraction where the "dirt" acts as a lens itself. Future research will focus on explicitly modeling these reflections and extending the framework to full-motion video to capture the dynamic nature of splashing and drying on the road.
Summary updated