Cross-Domain Image Classification through Neural-Style Transfer Data Augmentation

Yijie Xu, Arushi GoelarXiv 2019 · OpenReview

Abstract

In particular, the lack of sufficient amounts of domain-specific data can reduce the accuracy of a classifier. In this paper, we explore the effects of style transfer-based data transformation on the accuracy of a convolutional neural network classifiers in the context of automobile detection under adverse winter weather conditions. The detection of automobiles under highly adverse weather conditions is a difficult task as such conditions present large amounts of noise in each image. The InceptionV2 architecture is trained on a composite dataset, consisting of either normal car image dataset , a mixture of normal and style transferred car images, or a mixture of normal car images and those taken at blizzard conditions, at a ratio of 80:20. All three classifiers are then tested on a dataset of car images taken at blizzard conditions and on vehicle-free snow landscape images. We evaluate and contrast the effectiveness of each classifier upon each dataset, and discuss the strengths and weaknesses of style-transfer based approaches to data augmentation.

Introduction

Deep learning models are notoriously "data-hungry" and often suffer from domain bias. A classifier trained on thousands of crisp, sunny photos of cars will likely fail when it encounters a car obscured by a heavy snowstorm. This cross-domain problem—where the training domain (clean images) doesn't match the target domain (noisy, snowy images)—is a significant hurdle for robust computer vision.

Collecting high-quality data for specific adverse conditions is difficult, expensive, and increasingly unpredictable. While traditional data augmentation (like flipping or rotating images) helps, it doesn't account for complex environmental "noise" like snow or fog. This study investigates whether Neural Style Transfer (NST) can be used to "synthesize" this domain shift, providing a scalable alternative to manual data collection in the field.

Methodology

The researchers employed a workflow centered on Neural Style Transfer (NST), a technique that separates content features from style features and recombines them. In this study, "vanilla" car images served as content, while a reference image of a blizzard provided the style. Using a VGG-19 network, the authors created a "Styled" dataset where the geometry of the car remained intact, but the texture and "noise" of a blizzard were overlaid.

Visual comparison of real blizzard images (a, d), NST-styled images (b, e), and empty snowy landscapes (c).

An InceptionV2 architecture (with a ResNet34 backbone) was trained on three composite datasets to evaluate the impact of synthetic data:

  • Model A (Vanilla): 100% "vanilla" car images.
  • Model B (Styled): 80% vanilla images + 20% NST-generated "Styled" images.
  • Model C (Blizzard): 80% vanilla images + 20% real-world "Blizzard" images.

The models were evaluated against real-world blizzard images and a "Landscape" dataset consisting of empty, snowy scenes to measure the false-positive rate.

Results and Discussion

The results demonstrate that NST acts as an effective "domain-specific noise" generator. Unlike simple Gaussian noise or color shifts, NST introduces transformations that mimic visual physics—such as snow clinging to surfaces or blurring edges—while preserving the semantic "car-ness" of the image.

Quantitative results showing accuracy and false positive rates for the three models across different test sets.

As shown in the table above, Model A achieved 80.3% accuracy on real blizzard images. Incorporating just 20% NST-styled images (Model B) increased this to 88.8% and significantly reduced model uncertainty, dropping the standard deviation of error from 8.5% to 3.5%.

A critical finding was the "False Positive Trap." While Model C (trained on real blizzard data) had the highest raw accuracy (96.7%), it failed the "Landscape" test with a 23.6% false-positive rate. The model essentially learned to associate "snow" itself with the "car" class. In contrast, the NST-augmented Model B maintained a much lower false-positive rate of 8.6%, suggesting it learned to see through the noise rather than treating the noise as a primary feature. The authors also noted that the number of NST iterations is a critical hyperparameter; excessive iterations can lead to overly abstract images that eventually degrade classifier performance.

Conclusion

Neural Style Transfer offers a viable middle ground for cross-domain augmentation, improving generalization without the heavy false-positive penalties seen when training on limited real-world noisy data. While NST can introduce some blurriness that may obscure fine details, it remains a powerful tool for robustifying classifiers against environmental noise. Future work will explore more advanced generative models, such as GANs, to produce even more realistic synthetic weather effects.

Summary updated

Abstract

In particular, the lack of sufficient amounts of domain-specific data can reduce the accuracy of a classifier. In this paper, we explore the effects of style transfer-based data transformation on the accuracy of a convolutional neural network classifiers in the context of automobile detection under adverse winter weather conditions. The detection of automobiles under highly adverse weather conditions is a difficult task as such conditions present large amounts of noise in each image. The InceptionV2 architecture is trained on a composite dataset, consisting of either normal car image dataset , a mixture of normal and style transferred car images, or a mixture of normal car images and those taken at blizzard conditions, at a ratio of 80:20. All three classifiers are then tested on a dataset of car images taken at blizzard conditions and on vehicle-free snow landscape images. We evaluate and contrast the effectiveness of each classifier upon each dataset, and discuss the strengths and weaknesses of style-transfer based approaches to data augmentation.
BibTeX
@misc{xu2019cross,
  author = {Yijie Xu and Arushi Goel},
  title = {{Cross-Domain Image Classification through Neural-Style Transfer Data Augmentation}},
  year = {2019},
  eprint = {1910.05611},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.05611},
}
Figure