Translate the Facial Regions You Like Using Self-Adaptive Region Translation

Wenshuang Liu, Wenting Chen, Zhanjia Yang, Linlin Shen

Abstract

Facial image-to-image translation has traditionally struggled with "global leakage," where changing one attribute (like hair) inadvertently alters others (like skin tone). We present SART (Self-Adaptive Region Translation), a GAN-based framework designed for precise, region-level facial attribute transfer. By integrating a novel Region-adaptive Instance Normalization (RIN) technique and a specific Region Matching Loss (RML), SART enables surgical precision in facial editing, ensuring that style modifications in targeted areas do not distort the rest of the image.

Introduction

Most state-of-the-art image translation models, such as StarGAN or FUNIT, operate at the image level, treating the entire face as a single domain. Consequently, transformations like aging often apply globally, leading to unintended changes in static regions. While mask-based synthesis methods like SEAN or SPADE offer more control, they often struggle to preserve the fine-grained content and natural flow of the original image, forcing generated features into rigid mask shapes. SART bridges this gap by providing mask-based control while maintaining the identity and texture of the original subject.

Methodology

The SART architecture comprises a content encoder, a style encoder, and a decoder. The framework's precision stems from two primary innovations:

Region-adaptive Instance Normalization (RIN)

Unlike standard Instance Normalization (AdaIN) which applies a single style to an entire feature map, RIN utilizes a semantic mask to apply distinct style codes (gamma and beta parameters) to different regions. This allows the decoder to process multiple styles—such as a "surprised" eye expression and a "neutral" mouth—simultaneously in a single generation pass.

The SART generator architecture and the detailed RIN Res-Block structure.

Region Matching Loss (RML)

To ensure regional independence, the authors introduced the Region Matching Loss. This function compares two generated versions of the same face: one with all regions translated and one where only a specific $i$-th region is translated. RML forces the $i$-th region in both images to be identical while ensuring all other regions in the partially translated image match the original content. This prevents the network from taking "shortcuts" that modify the entire face to achieve local goals.

Experiments and Results

SART was evaluated on three diverse datasets: Morph (age-related translation), RaFD (expression translation), and CelebAMask-HQ (high-resolution attribute and gender translation).

Quantitative Performance

SART consistently outperformed StarGAN, SEAN, and FUNIT across key metrics:

  • Accuracy: On CelebAMask-HQ, SART achieved a translation accuracy of 97.06%, significantly higher than SEAN (72.95%) and StarGAN (62.17%).
  • Visual Realism: SART achieved the lowest Frechet Inception Distance (FID) scores, hitting 31.06 on CelebAMask-HQ, nearly half that of SEAN (61.06).

Qualitative Precision

Qualitatively, SART demonstrates the unique ability to modify specific regions—like changing eyes to an "angry" expression—while keeping other regions, such as a "happy" mouth, perfectly intact without boundary artifacts or ghosting.

Qualitative results demonstrating SART's ability to edit specific facial regions independently.

Conclusion and Future Work

SART establishes a new standard for localized facial editing by isolating style transformations through RIN and RML. While the model is currently dependent on the quality of input semantic masks, its ability to adapt both shape and texture naturally exceeds previous region-filling approaches. Future research may extend this framework to video, enabling localized expression editing and virtual makeup applications in motion.

Summary updated

Abstract

With the progression of Generative Adversarial Networks (GANs), image translation methods has achieved increasingly remarkable performance. However, most available methods can only achieve image level translation, which is unable to precisely control the regions to be translated. In this paper, we propose a novel self-adaptive region translation network (SART) for region-level translation, which uses region-adaptive instance normalization (RIN) and a region matching loss (RML) for this task. We first encode the style and content image for each region with style and content encoder. To translate both shape and texture of the target region, we inject region-adaptive style features into the decoder by RIN. To ensure independent translation among different regions, RML is proposed to measure the similarity between the non-translated/translated regions of content and translated images. Extensive experiments on three publicly available datasets, i.e. Morph, RaFD and CelebAMask-HQ, suggest that our approach demonstrate obvious improvement over state-of-the-art methods like StarGAN, SEAN and FUNIT. Our approach has further advantages in precise control of the regions to be translated. As a result, region level expression changes and step-by-step make-up can be achieved. The video demo is available at (https://youtu.be/DvIdmcR2LEc).
BibTeX
@inproceedings{liu2021translate,
  author = {Wenshuang Liu and Wenting Chen and Zhanjia Yang and Linlin Shen},
  title = {{Translate the Facial Regions You Like Using Self-Adaptive Region Translation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i3.16316},
}
Figure