Experiments on Anomaly Detection in Autonomous Driving by Forward-Backward Style Transfers

Daniel Bogdoll, Meng Zhang, Maximilian Nitsche, J. Marius Zöllner2022 · DOI · OpenReview

Abstract

Great progress has been achieved in the community of autonomous driving in the past few years. As a safety-critical problem, however, anomaly detection is a huge hurdle towards a large-scale deployment of autonomous vehicles in the real world. While many approaches, such as uncertainty estimation or segmentation-based image resynthesis, are extremely promising, there is more to be explored. Especially inspired by works on anomaly detection based on image resynthesis, we propose a novel approach for anomaly detection through style transfer. We leverage generative models to map an image from its original style domain of road traffic to an arbitrary one and back to generate pixelwise anomaly scores. However, our experiments have proven our hypothesis wrong, and we were unable to produce significant results. Nevertheless, we want to share our findings, so that others can learn from our experiments.

Introduction

For autonomous vehicles (AVs), the "open world" presents significant challenges. While deep learning models excel at identifying common objects like cars and pedestrians, they often struggle with "corner cases"—atypical objects or conditions not present in their training data, such as a dog running across a highway or lost cargo. Detecting these anomalies is a critical hurdle for safe AV deployment.

One popular strategy for addressing this is reconstructive detection. This approach assumes that a model trained only on "normal" road scenes will fail to accurately rebuild an image containing an anomaly. This paper explores whether style transfer can serve as a reconstructive bottleneck. The hypothesis is that by converting a road scene into an arbitrary artistic style and then back to the road domain, anomalous objects would either disappear or distort significantly, allowing them to be caught via pixel-wise comparison.

Methodology

The proposed pipeline consists of three main stages:

  1. Forward Style Transfer: An input image from the road domain is converted into an arbitrary artistic style (e.g., Van Gogh’s The Starry Night).
  2. Backward Style Transfer: The stylized image is converted back into the original road traffic domain.
  3. Anomaly Mapping: A pixel-by-pixel difference map is calculated between the original and the reconstructed image to highlight potential anomalies.
The proposed forward-backward style transfer pipeline for generating pixelwise anomaly scores.

The authors tested two primary technical approaches: CycleGAN, for unpaired image-to-image translation, and Neural Style Transfer (NST), an optimization-based approach using a pre-trained VGG19 network. The core idea relied on the Manifold Hypothesis: the assumption that "normal" samples live on a specific mathematical manifold. By forcing an image through successive style changes, the authors hoped the model would "project" the image back onto the "normal" manifold, effectively erasing out-of-distribution elements.

Experimental Setup

The study utilized standard benchmarks for training and evaluation:

  • Normal Data: KITTI and Cityscapes datasets provided standard urban and highway driving footage.
  • Anomaly Evaluation: Fishyscapes and Lost and Found datasets were used, featuring out-of-distribution objects like animals and road hazards.

Experimental Results

The experiments yielded definitively negative results. The most significant insight was the Robustness Trap: modern generative models are actually too robust. Even when an image was heavily stylized, the underlying geometric structures of anomalies were preserved. Consequently, the models reconstructed "unknown" objects with high fidelity during the backward transfer.

Qualitative results showing that anomalies (red boxes) are faithfully reconstructed, leading to noisy anomaly maps that fail to highlight out-of-distribution objects.

Visual inspections of the resulting anomaly maps showed that objects like dogs or cargo were reconstructed almost as clearly as the road and trees. The difference maps were dominated by pixel-wise noise and artifacts from the style transfer process itself, rather than clear signals at the anomaly locations. While NST allowed for control over stylization degree, increasing the style weight enough to potentially "erase" an anomaly resulted in the total loss of the scene's content. CycleGAN proved to be an excellent reconstructor, which, while usually a goal in GAN research, directly caused the failure of this anomaly detection hypothesis.

Conclusion

The primary limitation of this approach is the high reconstructive accuracy of current generative models. Because these models preserve content (edges, shapes, and positions) so effectively during style transfer, the anomalies are never lost. The authors suggest that a more constrained bottleneck, perhaps involving semantic segmentation as an intermediate step, might be necessary. By publishing these negative results, the authors aim to prevent others from pursuing this specific dead end and emphasize the importance of sharing failures in scientific literature.

Summary updated

Abstract

Great progress has been achieved in the community of autonomous driving in the past few years. As a safety-critical problem, however, anomaly detection is a huge hurdle towards a large-scale deployment of autonomous vehicles in the real world. While many approaches, such as uncertainty estimation or segmentation-based image resynthesis, are extremely promising, there is more to be explored. Especially inspired by works on anomaly detection based on image resynthesis, we propose a novel approach for anomaly detection through style transfer. We leverage generative models to map an image from its original style domain of road traffic to an arbitrary one and back to generate pixelwise anomaly scores. However, our experiments have proven our hypothesis wrong, and we were unable to produce significant results. Nevertheless, we want to share our findings, so that others can learn from our experiments.
BibTeX
@inproceedings{bogdoll2022experiments,
  author = {Daniel Bogdoll and Meng Zhang and Maximilian Nitsche and J. Marius Zöllner},
  title = {{Experiments on Anomaly Detection in Autonomous Driving by Forward-Backward Style Transfers}},
  booktitle = {2022},
  year = {2022},
  doi = {10.1109/iceccme55909.2022.9988287},
}
Figure