Style transfer for headshot portraits

Yichang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, Frédo DurandACM Trans. Graphics 2014 · DOI

TL;DR

This paper introduces a novel multiscale approach for transferring the artistic style of one headshot portrait to another. It addresses the limitations of existing methods by focusing on the specific, local retouching techniques used by professional photographers for headshots, enabling users to easily replicate desired looks.

What Problem Does This Paper Address?

Professional photographers meticulously edit headshot portraits to achieve compelling visual styles, a process that requires advanced skills and is difficult to automate. Generic image stylization algorithms often fail on headshots because they don't account for the fine-grained, local adjustments (e.g., skin texture, hair rendition, lighting) that are crucial for a polished portrait. This paper aims to bridge this gap by providing a technique that can transfer the specific photographic style of an example headshot to a new portrait, preserving the subject's identity while adopting the desired aesthetic.

Methods

The core of the proposed method is a multiscale local transfer technique. It begins by establishing a dense correspondence between the input and example headshot images using a three-step process, including SIFT Flow for refinement. This ensures that corresponding facial features are aligned.

The key innovation lies in the multiscale decomposition of both images into Laplacian stacks. For each scale (or "band"), the algorithm computes a local energy map that captures the intensity variations. The transfer then involves matching these local statistics across scales. Specifically, it calculates a "gain map" for each scale, which dictates how to modify the input's local contrast to match the example's. This gain map is applied to the input's Laplacian stack to create a new stack.

To handle potential artifacts caused by mismatches (e.g., a mole in the example but not the input, or glasses), a robust gain map is introduced, which clamps and smooths the gain values. The method also incorporates postprocessing steps to transfer eye highlights and replace the background, further enhancing the realism of the stylized portrait. For situations where a photographer has a collection of consistently styled headshots, an automatic example selection algorithm is provided to find the best reference image for a given input.

Datasets

The paper uses portrait collections from three photographers, each containing 50-75 headshots. They also tested their method on 94 photos collected from Flickr, which presented challenges due to uncontrolled lighting, noise, and cluttered backgrounds. No new benchmarks were introduced.

Key Insights and Ideas

  • Headshot stylization is fundamentally local: Unlike general image stylization, headshots require precise, localized adjustments to features like skin, hair, and lighting. Global methods are insufficient.
  • Multiscale analysis is crucial: Capturing the nuances of photographic style, from fine skin textures to broader lighting patterns, necessitates analyzing image statistics at various scales.
  • Local energy maps as style descriptors: The concept of using local energy maps derived from Laplacian pyramid decompositions effectively captures the "texture" and "contrast" characteristics of a style.
  • Robustness to outliers is key: Direct gain mapping can lead to artifacts when the input and example differ significantly in certain regions. A robust gain map mechanism is essential for producing plausible results.
  • Automatic example selection simplifies workflow: For photographers with consistent styles, automating the selection of the best reference image significantly improves usability.

Key Results

The method successfully transfers various photographic styles, including high-contrast black-and-white, warm and soft lighting, and low-key styles. It works on both indoor and outdoor photos, and for subjects of different genders. The authors demonstrate that their multiscale approach outperforms single- or two-scale methods. They also show that their method can handle some amount of illumination change and can be extended to video with temporal coherence. The running time for their unoptimized MATLAB implementation is around 12 seconds for a 1300x900 image.

Limitations and Future Work

Limitations: * The quality of the output is often limited by the quality of the matting mask used for separating the face from the background. * The method can magnify input noise. * Selecting an example that closely matches the input's facial attributes (e.g., beard, skin color) is important for natural results; mismatched attributes can lead to unnatural outputs. * The method cannot create or remove hard shadows. * Some identity transfer from the example can occur, leading to artifacts. * The face detector fails on profile headshots. * The method is not designed for non-photorealistic styles (e.g., cartoons, paintings). * Eye highlight transfer can fail if the input and example have very different eye colors.

Future Work: * Exploring style transfer from multiple examples, potentially combining different facial regions from different sources to achieve better matches. * Improving the robustness of the method to handle more extreme variations in input data and lighting.

Summary updated

BibTeX
@article{shih2014style,
  author = {Yichang Shih and Sylvain Paris and Connelly Barnes and William T. Freeman and Frédo Durand},
  title = {{Style transfer for headshot portraits}},
  journal = {ACM Trans. Graphics},
  year = {2014},
  doi = {10.1145/2601097.2601137},
}
Figure