Semantic-Aware Generation of Multi-View Portrait Drawings

Biao Ma, Fei Gao, Chang Jiang, Nannan Wang, Gang Xu

Abstract

Generating 3D-consistent artistic portraits remains a significant challenge for traditional Neural Radiance Fields (NeRF) and 3D-aware GANs. These models often fail in non-photorealistic domains because artistic strokes are inherently view-inconsistent and drawing datasets lack the dense structural cues found in photography. We present SAGE (Semantic-Aware GEnerator), a framework that leverages facial semantic maps as a view-consistent bridge to synthesize high-quality portrait drawings across multiple viewpoints. By decoupling geometric structure from artistic style through a two-stage training strategy, SAGE achieves superior consistency and visual quality compared to existing state-of-the-art models.

Introduction

Recent progress in 3D-aware generative models has enabled the synthesis of photorealistic human faces with high multi-view consistency. However, these models often struggle when applied to artistic domains such as pen, pencil, or line art. The authors identify three primary obstacles: the violation of the "consistency assumption" (where artistic strokes change or disappear depending on the angle), the sparsity of 3D structural information in drawings, and the extreme scarcity of high-quality multi-view artistic datasets.

SAGE addresses these challenges by shifting the focus from radiance to semantics. While the specific placement of a pencil stroke may be view-dependent, the underlying facial semantics—the location of the eyes, nose, and mouth—remain consistent in 3D space. By using these semantic labels to guide the drawing process, SAGE maintains structural integrity even during complex head rotations.

Methodology

The SAGE framework utilizes a two-stage training strategy designed to inherit 3D priors from photorealistic data before translating them into artistic styles.

Stage 1: Geometric Scaffolding

Because it is nearly impossible to learn accurate 3D geometry from sparse line art alone, the model is first trained on large datasets of human photos (CelebAMask-HQ). This stage allows the NeRF module ($G_F$) to learn the underlying 3D geometry of the human head and its corresponding semantic mapping.

Stage 2: Artistic Translation

In the second stage, the model introduces a Semantic-aware Domain Translator ($G_p$). This module takes the 3D features generated by the NeRF backbone and translates them into a specific artistic style. To ensure the strokes align with the facial structure, the translator employs Spatially-Adaptive Normalization (SPADE), which conditions the style generation on the facial semantic map.

To further combat data scarcity, the researchers used a modified 2D image-to-image translation model to create a large synthetic dataset of drawings from photos, which served as additional training material for the 3D generator.

Experiments and Results

The authors evaluated SAGE across several diverse artistic styles, including pen-drawings (APDrawing), pencil-drawings (FS2K), and oil paintings (WikiArt).

Quantitative Performance

SAGE was compared against state-of-the-art (SOTA) models such as MVCGAN and CIPS-3D. SAGE achieved significantly lower scores in SIFID (Single Image Frechet Inception Distance) and SWD (Sliced Wasserstein Distance), indicating higher image quality and stylistic alignment. For example, in the pen-drawing category, SAGE achieved a SIFID of 3.07, outperforming CIPS-3D (3.51) and MVCGAN (4.19).

Qualitative Consistency

Qualitatively, SAGE demonstrated a marked improvement in multi-view stability. While baseline models often produced "melting" artifacts or inconsistent ink patterns when the viewpoint rotated, SAGE maintained clean, distinct facial structures. This was reflected in a user study with 37 participants, where over 92% of respondents preferred SAGE’s results over the baselines across all tested styles.

Discussion and Future Work

SAGE demonstrates that in non-photorealistic rendering, maintaining the "logic" of the face through semantics is more critical for 3D consistency than maintaining pixel-perfect radiance. By decoupling content (geometry) from style (translator), the framework effectively bridges the gap between 2D artistry and 3D-aware generation.

Currently, SAGE generates random 3D-consistent faces. Future research directions include incorporating GAN-inversion to allow the generation of multi-view drawings from a single user-provided photo. Additionally, the authors aim to explore few-shot learning techniques to reduce the amount of data required to master a new artistic style.

Summary updated

BibTeX
@inproceedings{ma2023semantic,
  author = {Biao Ma and Fei Gao and Chang Jiang and Nannan Wang and Gang Xu},
  title = {{Semantic-Aware Generation of Multi-View Portrait Drawings}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/140},
}
Figure