Invertible Conditional GAN Revisited: Photo-to-Manga Face Translation with Modern Architectures (Student Abstract)

Taro Hatakeyama, Ryusuke Saito, Komei Hiruta, Atsushi Hashimoto, Satoshi KuriharaAAAI 2023 · DOI

TL;DR

This paper introduces IcFGAN, a novel approach for translating realistic photos into manga-style faces while preserving identity. It achieves this by combining modern GAN inversion techniques with a conditional GAN architecture, enabling significant style transformations without requiring complex feature disentanglement.

What Problem Does This Paper Address?

Translating images between vastly different styles, like realistic photos to manga, is challenging. Manga faces often involve drastic geometric exaggerations and a simplified, line-based aesthetic that contrasts sharply with the detailed, photorealistic nature of input images. Existing methods struggle to achieve this level of transformation while simultaneously preserving the original subject's identity. Many are limited to more subtle style transfers or require paired training data, which is often unavailable for such extreme style shifts. This paper aims to bridge this gap by enabling unpaired photo-to-manga face translation with high fidelity in both style and identity.

Methods

The core of IcFGAN lies in its two-step process:

  1. Optimization-Based Inversion: This step takes a real photo and "inverts" it into a latent representation that can be understood by a GAN. Crucially, this inversion process is guided by the original image's style (represented by a condition code) and aims to find a latent code that, when decoded by the GAN, reconstructs the input image. This is achieved by minimizing a loss function that includes both perceptual similarity (LPIPS) and pixel-wise reconstruction error.
  2. Condition Switch Generation: Once the latent code is obtained, the condition code is switched from the original photo style to the target manga style. This modified latent code is then fed into a pre-trained conditional GAN (cFGAN) to generate the manga-style face. The cFGAN used is an extension of FastGAN, which is known for its efficiency and ability to generate high-quality images with fewer parameters than models like StyleGAN2. This efficiency is key to adapting features across different conditions and preserving identity.

The novelty lies in revisiting the concept of invertible GANs (like IcGAN) but updating it with modern GAN inversion techniques and a more efficient conditional GAN architecture (cFGAN), specifically tailored for significant style gaps.

Datasets

  • Training:
    • MUCT Face Database: Used for input real photos.
  • Tezuka Dataset: Used for target manga faces. This dataset is noted as being "landmark-less," which poses a challenge for some traditional alignment-based methods.
  • Evaluation: The paper uses the MUCT and Tezuka datasets for qualitative and quantitative evaluations.

Key Insights and Ideas

  • Revisiting Invertible GANs: The paper demonstrates that older concepts like invertible GANs, when combined with modern advancements, can still be highly effective for challenging translation tasks.
  • Bridging Style Gaps: IcFGAN's ability to handle significant geometric and textural differences (photo to manga) without explicit feature disentanglement is a key breakthrough. This suggests that a well-designed inversion and generation process can implicitly manage these complex transformations.
  • Efficiency and Identity Preservation: Using a cFGAN based on FastGAN allows for efficient generation and, importantly, better preservation of facial identity even under drastic style changes. This is attributed to the model's fewer parameters and improved feature adaptation.

Key Results

  • Qualitative: IcFGAN successfully translates realistic faces into manga style, preserving identity while achieving significant geometric and stylistic changes. Visual comparisons show it outperforms state-of-the-art methods in maintaining identity and achieving the target manga aesthetic.
  • Quantitative:
    • Fréchet Inception Distance (FID): IcFGAN achieved the best FID score (33.15) among the evaluated methods, indicating that the generated manga faces are statistically closer to the real Tezuka Dataset distribution.
  • User Studies: In user studies, IcFGAN performed best:
  • User Study 1 (manga style similarity): 51.16% * User Study 2 (identity preservation): 64.60%

Limitations and Future Work

  • Limitations: The paper acknowledges that while IcFGAN performs well, achieving perfect identity preservation and style translation simultaneously under extreme style gaps remains a challenge. The "landmark-less" nature of the Tezuka dataset also presents inherent difficulties.
  • Future Work: Not explicitly detailed in the provided text, but the success of IcFGAN suggests future work could explore applying this approach to other extreme style translation tasks or further refining the GAN inversion and generation architectures for even greater fidelity.

Summary updated

BibTeX
@inproceedings{hatakeyama2023invertible,
  author = {Taro Hatakeyama and Ryusuke Saito and Komei Hiruta and Atsushi Hashimoto and Satoshi Kurihara},
  title = {{Invertible Conditional GAN Revisited: Photo-to-Manga Face Translation with Modern Architectures (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26972},
}
Figure