Learning Portrait Style Representations

Sadat Shaik, Bernadette Bucher, Nephele Agrafiotis, Stephen Phillips, Kostas Daniilidis, William SchmennerarXiv 2020 · arXiv

Abstract

Traditional computer vision approaches often reduce artistic "style" to local textures or brushstrokes. This paper bridges the gap between computational methods and art history by developing style representations for portrait paintings that incorporate expert human intuition. By combining Variational Autoencoders (VAEs) and VGG-16 architectures with "triplet labels" curated by art historians, the authors create a model that aligns computational embeddings with sophisticated stylistic concepts. The resulting framework enables advanced tasks such as zero-shot artist classification and nuanced style interpolation.

Introduction: Redefining Style in Computer Vision

In most computer vision research, style is treated as a synonym for local texture—exemplified by Neural Style Transfer techniques that overlay patterns onto digital images. However, art historians argue that style is far more complex: it is historically bounded, influenced by materials, and inextricably linked to content. For instance, an artist's choice of subject or composition is as much a stylistic decision as their brushwork.

The authors identify two primary hurdles in the field:

  1. The Definition Gap: Existing computational methods do not align with professional art historical definitions.
  2. Data Scarcity: There is a lack of large-scale, high-quality datasets curated for fine-grained artistic analysis rather than broad genre classification.

Methods: Integrating Expert Knowledge

The researchers investigated how different "priors" (pre-existing knowledge or constraints) affect a model's ability to learn style. They tested four primary configurations:

  • Statistical Prior (VAE): A Variational Autoencoder trained on portraits to learn the underlying data distribution.
  • Photo-realism Prior (VGG-16): A model pretrained on ImageNet to leverage its ability to recognize low-level features like edges and colors.
  • Expert Human Knowledge Prior (Triplet Loss): The core innovation involves a triplet loss function. An art historian evaluates three images—an "anchor," a "positive" (stylistically similar), and a "negative" (stylistically different)—guiding the model to align its embeddings with expert similarity judgments.
  • Hybrid Models: These combine the VGG or VAE base with the expert triplet loss to create a more robust representation.
Architecture of the hybrid model combining VAE, VGG-16, and Triplet Loss.

By training with triplet loss, the model is forced to map stylistically similar paintings closer together in a high-dimensional "style space" (embedding), while pushing dissimilar ones further apart.

Datasets and Expert Annotation

To support this research, the authors introduced The Portrait Dataset, consisting of 6,886 portrait images scraped from Artstor. The dataset includes cleaned labels for artists and time periods. Crucially, they developed a custom web tool for professional art historians to provide the similarity labels required for the triplet loss training. Both the dataset and the labeling tool have been made publicly available.

Key Insights and Results

Beyond the Brushstroke

The approach captures higher-level characteristics like pose, lighting, and composition. This is particularly effective for artists like Jan van Eyck, who intentionally obscured their brushstrokes, rendering texture-only models ineffective.

The Power of Photo-Priors and Focus Shifting

Surprisingly, models pretrained on photographs (ImageNet) provide a strong foundation for art analysis. Using Grad-CAM activation maps, the authors demonstrated that adding expert triplet loss shifted the model's focus. While a standard VGG model might focus on background elements, the expert-trained model shifted its attention to the subject's pose and garment textures—features art historians actually use to distinguish styles.

Performance Metrics

  • Visual Clustering: t-SNE projections showed that the VGG + Triplet Loss model created the most coherent clusters, successfully grouping works by specific artists while distinguishing between subtle differences in presentation.
  • Zero-Shot Artist Classification: The VGG + Triplet model achieved the highest accuracy in identifying artists the model had not seen during training. While absolute accuracy was approximately 7-8%, this is significantly higher than the random baseline of 0.106% across 943 artist classes.
  • Style Interpolation: Using the VAE's continuous latent space, the authors demonstrated "walking" between the styles of different artists (e.g., Rembrandt to Van Gogh), generating intermediate portraits that blend stylistic traits.

Conclusion and Future Work

While currently focused on portraiture, this work demonstrates the potential of aligning deep learning with domain-specific expertise. Future research aims to expand the dataset to include landscapes and still lifes, and to map continuous style representations to discrete art historical concepts, such as the Wölfflin principles. The authors also suggest exploring more advanced generative architectures to further refine the synthesis of artistic style.

Summary updated

Abstract

Style analysis of artwork in computer vision predominantly focuses on achieving results in target image generation through optimizing understanding of low level style characteristics such as brush strokes. However, fundamentally different techniques are required to computationally understand and control qualities of art which incorporate higher level style characteristics. We study style representations learned by neural network architectures incorporating these higher level characteristics. We find variation in learned style features from incorporating triplets annotated by art historians as supervision for style similarity. Networks leveraging statistical priors or pretrained on photo collections such as ImageNet can also derive useful visual representations of artwork. We align the impact of these expert human knowledge, statistical, and photo realism priors on style representations with art historical research and use these representations to perform zero-shot classification of artists. To facilitate this work, we also present the first large-scale dataset of portraits prepared for computational analysis.
BibTeX
@misc{shaik2020learning,
  author = {Sadat Shaik and Bernadette Bucher and Nephele Agrafiotis and Stephen Phillips and Kostas Daniilidis and William Schmenner},
  title = {{Learning Portrait Style Representations}},
  year = {2020},
  eprint = {2012.04153},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.04153},
}
Figure