A Learned Representation For Artistic Style

Vincent Dumoulin, Jonathon Shlens, Manjunath Kudlur

Abstract: The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general. In this work we investigate the construction of a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings. We demonstrate that such a network generalizes across a diversity of artistic styles by reducing a painting to a point in an embedding space. Importantly, this model permits a user to explore new painting styles by arbitrarily combining the styles learned from individual paintings. We hope that this work provides a useful step towards building rich models of paintings and offers a window on to the structure of the learned representation of artistic style.

The Bottleneck of Single-Style Networks

Artistic style transfer—the process of redrawing a "content" image in the "style" of a specific painting—has evolved rapidly in recent years. Early methods relied on slow optimization processes that took minutes to generate a single image. While subsequent "feedforward" networks made this process instantaneous, they introduced a significant bottleneck: one network could only learn one style.

Starry Night Style

If a developer wanted to offer 100 different artistic filters in a mobile application, they would need to store 100 separate, memory-intensive models. This brute-force approach ignores a fundamental intuition about art: different paintings often share common visual elements, such as brushstrokes, textures, and geometric shapes. Researchers at Google Brain tackled this inefficiency by developing a single, scalable network capable of parsimoniously capturing the "visual vocabulary" of dozens of different styles simultaneously.

The Solution: Conditional Instance Normalization

The authors introduced a novel layer modification called Conditional Instance Normalization (CIN).

In standard style transfer networks, Instance Normalization (IN) is used to normalize the mean and variance of feature maps, helping the network abstract away specific contrast or lighting in the input image. The researchers discovered a surprising fact: to completely change the artistic style of the output, one does not need to alter the heavy convolutional weights of the network. Instead, one only needs to adjust the scaling ($\gamma$) and shifting ($\beta$) parameters applied after normalization.

In a CIN layer, for a given style $s$, the activation $x$ is transformed using the following equation: $$z = \gamma_s \left( \frac{x - \mu}{\sigma} \right) + \beta_s$$ Where $\mu$ and $\sigma$ are the mean and standard deviation of the activations. While the millions of convolutional weights are shared across all styles, each specific style is assigned its own unique pair of $\gamma$ and $\beta$ vectors. This architecture allows the model to instantly "switch" styles simply by swapping out these tiny parameter sets.

Training on a Diverse Visual Vocabulary

To prove the model's versatility, it was trained using the ImageNet dataset as a diverse corpus of real-world content photographs. For the style targets, the authors curated a collection of 32 varied paintings.

Pop Art Style

This collection spanned a wide array of artistic movements, ranging from the Impressionism of Claude Monet to the bold Pop Art of Roy Lichtenstein and the Expressionism of Edvard Munch. The model was subsequently evaluated on standard photographs, including landscapes like the Golden Gate Bridge and various portraits, successfully applying these diverse styles.

Key Insights and Performance

The implementation of CIN yielded several profound insights into how neural networks represent style:

  • Style as an Affine Transformation: The "essence" of an artistic style appears to be largely encoded in the mean and variance of the feature maps. By simply shifting and scaling these normalized features, the network can completely reconfigure its artistic output.
  • Extreme Parameter Efficiency: In the researchers' typical setup, the base model contains roughly 1.6 million parameters. However, only about 3,000 of those parameters (a mere 0.2%) are unique to each style. Adding a 33rd style to the model is incredibly memory-efficient.
  • Style Interpolation: Because styles are represented by simple vectors ($\gamma$ and $\beta$), they act as points in an embedding space. By taking a weighted average of the parameters for two different styles, the network produces a smooth, visually pleasing interpolation. A user can literally "dial in" a blend of 50% Van Gogh and 50% Monet in real-time.

Ultimately, a single network successfully learned 32 distinct styles, producing qualitative results indistinguishable from 32 separate, individually trained networks. Furthermore, when adding a new style to this pre-trained multi-style network, the model converged significantly faster than training from scratch (5,000 steps versus 40,000 steps).

Future Directions

While the method works brilliantly, the authors note that the underlying "why" remains a hypothesis. It is not yet strictly proven why the shared convolutional weights act as a universal vocabulary of style elements, or why the scaling factors perfectly act as inhibitors or enhancers of those specific elements.

Future work may explore "predictive style transfer." The current model requires an explicit style index as an input. A future iteration could involve a secondary "style-prediction" network that analyzes a novel painting and automatically generates the appropriate $\gamma$ and $\beta$ parameters, enabling true "zero-shot" style transfer for art the model has never seen before.

Summary updated

Abstract

The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general. In this work we investigate the construction of a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings. We demonstrate that such a network generalizes across a diversity of artistic styles by reducing a painting to a point in an embedding space. Importantly, this model permits a user to explore new painting styles by arbitrarily combining the styles learned from individual paintings. We hope that this work provides a useful step towards building rich models of paintings and offers a window on to the structure of the learned representation of artistic style.
BibTeX
@inproceedings{dumoulin2017learned,
  author = {Vincent Dumoulin and Jonathon Shlens and Manjunath Kudlur},
  title = {{A Learned Representation For Artistic Style}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=BJO-BuT1g},
}
Figure