What Can Style Transfer and Paintings Do For Model Robustness?

Hubert Lin, Mitchell van Zuijlen, Sylvia C. Pont, Maarten W. A. Wijntjes, Kavita Bala

Introduction

Deep learning models are notoriously brittle. A model trained on sunny photos might fail in the rain, or a slight change in camera angle might cause it to misclassify an object. To fix this, researchers often use data augmentation—artificially expanding the training set with flipped, rotated, or color-shifted images.

Recently, "style transfer" (applying the texture of a painting to a photo) has become a popular tool for teaching models to ignore textures and focus on shapes. However, the authors of this paper ask a deeper question: Is a stylized photo actually the same as a real painting? Artists don't just apply a "filter"; they selectively emphasize salient features and ignore irrelevant details. The paper investigates whether these human-centric artistic choices can teach models something that automated style transfer cannot.

Methodology

The researchers explored several hypotheses using a variety of style transfer techniques, including:

  • Arbitrary Style Transfer: Methods like AdaIN, ETNet, and TPFR that can apply any style to any image.
  • Learned Artistic Style Transfer (SACL): A method that learns the specific styles of various artists.
  • Intradomain Stylization: A novel experiment where they stylized photos using other photos as the style source, rather than paintings.
Overview of the style transfer framework and the resulting model invariances.

To evaluate robustness, they used the ImageNet-C framework, which tests models against 15 types of corruptions (noise, blur, weather, and digital artifacts). They also tested Out-of-Distribution (OOD) performance, looking at how well models generalize to entirely different datasets with different viewpoints and lighting.

The study utilized two primary benchmarks: the PACS domain generalization dataset and Materials, a large-scale dataset (120K images) newly constructed by the authors combining photograph patches with a large-scale painting dataset.

Key Insights

  • Diversity Trumps "Art": For standard style transfer augmentation, the "artistic" nature of the style image isn't the secret sauce. Stylizing a photo with another random photo (intradomain stylization) works just as well as using a Van Gogh, provided the style source is sufficiently different from the content.
  • Paintings as Perceptual Filters: Real paintings act as "perceptual data augmentation." Because artists emphasize what is human-recognizable and abstract away the rest, training on real paintings helps models focus on the most important semantic cues.
  • Complementary Invariances: Style transfer and real paintings solve different problems. Style transfer provides robustness against noise and digital corruptions, while real paintings significantly improve robustness to viewpoint and lighting changes.

Key Results

In a striking contrast, while style transfer often harmed performance on out-of-distribution images (viewpoint/lighting shifts), training with real paintings consistently improved it. Combining learned style transfer (SACL) with real paintings yielded the highest overall robustness. On the Materials dataset, this combination achieved a mean corruption accuracy of 51.49%, compared to 48.03% for a model trained only on photos.

Impact of painting percentage on mean corruption accuracy across PACS and Materials datasets.

Furthermore, the authors found that if you have a fixed budget for data annotation, allocating up to 50% of that budget to paintings (rather than just more photos) results in a more robust model. However, artform matters: sparse representations like line sketches actually reduced model robustness because they lacked the realistic shading and cues necessary for feature learning.

Conclusion

While current style transfer models still don't fully capture the "essence" of artistic creation, this work demonstrates that real-world artistic data provides unique benefits for model robustness. Future research will explore how coarser abstractions can be leveraged to teach models a hierarchy of invariances and develop style transfer methods that are more "perceptually grounded" to mimic human artistic emphasis.

Summary updated

BibTeX
@inproceedings{lin2021what,
  author = {Hubert Lin and Mitchell van Zuijlen and Sylvia C. Pont and Maarten W. A. Wijntjes and Kavita Bala},
  title = {{What Can Style Transfer and Paintings Do For Model Robustness?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=RSJzRvUriH1},
}
Figure