Abstract
Facial image-to-image translation has traditionally struggled with "global leakage," where changing one attribute (like hair) inadvertently alters others (like skin tone). We present SART (Self-Adaptive Region Translation), a GAN-based framework designed for precise, region-level facial attribute transfer. By integrating a novel Region-adaptive Instance Normalization (RIN) technique and a specific Region Matching Loss (RML), SART enables surgical precision in facial editing, ensuring that style modifications in targeted areas do not distort the rest of the image.
Introduction
Most state-of-the-art image translation models, such as StarGAN or FUNIT, operate at the image level, treating the entire face as a single domain. Consequently, transformations like aging often apply globally, leading to unintended changes in static regions. While mask-based synthesis methods like SEAN or SPADE offer more control, they often struggle to preserve the fine-grained content and natural flow of the original image, forcing generated features into rigid mask shapes. SART bridges this gap by providing mask-based control while maintaining the identity and texture of the original subject.
Methodology
The SART architecture comprises a content encoder, a style encoder, and a decoder. The framework's precision stems from two primary innovations:
Region-adaptive Instance Normalization (RIN)
Unlike standard Instance Normalization (AdaIN) which applies a single style to an entire feature map, RIN utilizes a semantic mask to apply distinct style codes (gamma and beta parameters) to different regions. This allows the decoder to process multiple styles—such as a "surprised" eye expression and a "neutral" mouth—simultaneously in a single generation pass.
Region Matching Loss (RML)
To ensure regional independence, the authors introduced the Region Matching Loss. This function compares two generated versions of the same face: one with all regions translated and one where only a specific $i$-th region is translated. RML forces the $i$-th region in both images to be identical while ensuring all other regions in the partially translated image match the original content. This prevents the network from taking "shortcuts" that modify the entire face to achieve local goals.
Experiments and Results
SART was evaluated on three diverse datasets: Morph (age-related translation), RaFD (expression translation), and CelebAMask-HQ (high-resolution attribute and gender translation).
Quantitative Performance
SART consistently outperformed StarGAN, SEAN, and FUNIT across key metrics:
- Accuracy: On CelebAMask-HQ, SART achieved a translation accuracy of 97.06%, significantly higher than SEAN (72.95%) and StarGAN (62.17%).
- Visual Realism: SART achieved the lowest Frechet Inception Distance (FID) scores, hitting 31.06 on CelebAMask-HQ, nearly half that of SEAN (61.06).
Qualitative Precision
Qualitatively, SART demonstrates the unique ability to modify specific regions—like changing eyes to an "angry" expression—while keeping other regions, such as a "happy" mouth, perfectly intact without boundary artifacts or ghosting.
Conclusion and Future Work
SART establishes a new standard for localized facial editing by isolating style transformations through RIN and RML. While the model is currently dependent on the quality of input semantic masks, its ability to adapt both shape and texture naturally exceeds previous region-filling approaches. Future research may extend this framework to video, enabling localized expression editing and virtual makeup applications in motion.
Summary updated