MegaPortrait: Revisiting Diffusion Control for High-fidelity Portrait Generation

Han Yang, Sotiris Anagnostidis, Enis Simsar, Thomas HofmannarXiv 2024 · DOI · arXiv

Introduction

Personalized portrait generation has long struggled with the "identity-style dilemma"—a fundamental trade-off between maintaining the recognizable features of a subject (identity preservation) and accurately adopting the aesthetic qualities of a target environment (artistic adaptation). Existing diffusion-based models often fail by either washing out the subject's identity to match the style or producing "pasted-on" faces that clash with the background's lighting and geometry. MegaPortrait addresses this by decoupling facial geometry from shading through a novel "split-and-merge" pipeline.

Methodology: The Split-and-Merge Pipeline

MegaPortrait is built upon Stable Diffusion V1.5 and ControlNet, utilizing a modular architecture to handle different aspects of the portrait generation process.

Decoupling via Frequency

The core innovation lies in the realization that identity is primarily encoded in high-frequency (HF) geometric details, while style and lighting are represented in low-frequency (LF) shading information. The system employs three specialized networks:

  1. Identity Net: Fine-tuned via LoRA on a small set of source images, this module generates HF-Maps (capturing geometry) and Light Maps (capturing shading).
  2. Shading Net: This network uses the HF-Maps and Light Maps as control signals to synthesize a face that maintains the source identity but adopts the reference scene's lighting.
  3. Harmonization Net: This final stage ensures a seamless transition between the generated face and the reference image's body and background, correcting color grading and edge artifacts.
High-fidelity facial detail preservation achieved through the Identity Net and HF-Map extraction.

Gaze Correction

To solve the "dead eyes" problem common in overfitted personalized models, MegaPortrait introduces a "free-lunch" gaze correction technique. By applying a local mask over the eyes and using classifier-free guidance with text prompts like "looking at the camera," the model restores expressive, vivid gazes without requiring additional training or complex architectural changes.

Experimental Results

The model was evaluated on a curated benchmark of 100 unique identity-style pairs. Quantitative analysis using ArcFace (for identity similarity) and the Aesthetic Score Predictor V2 shows that MegaPortrait outperforms leading tools like Remini and IP-Adapter.

Model Aesthetic Score ID Score
MegaPortrait 6.3789 0.6630
Remini 6.2431 0.6157
IP-Adapter-FaceID 6.1062 0.6580

Qualitatively, MegaPortrait excels in preserving the "soul" of the identity even under extreme lighting or non-photorealistic artistic styles, such as oil paintings or dramatic noir photography.

MegaPortrait demonstrates superior identity preservation across diverse reference styles and lighting conditions.

Limitations and Future Directions

While MegaPortrait sets a new standard for high-fidelity portraits, some challenges remain. The model occasionally inherits hair or eye colors from the reference style rather than the source identity. Furthermore, the authors suggest that future iterations could benefit from "aesthetic mining"—extracting user preferences from edited portraits—and exploring non-spatial ControlNets to better decouple facial expressions from identity.

Conclusion

By revisiting diffusion control through the lens of frequency decoupling, MegaPortrait effectively resolves the identity-style conflict. Its modular approach allows for high-fidelity, personalized generation that respects both the subject's unique features and the artistic integrity of the reference environment.

Summary updated

Abstract

We propose MegaPortrait. It's an innovative system for creating personalized portrait images in computer vision. It has three modules: Identity Net, Shading Net, and Harmonization Net. Identity Net generates learned identity using a customized model fine-tuned with source images. Shading Net re-renders portraits using extracted representations. Harmonization Net fuses pasted faces and the reference image's body for coherent results. Our approach with off-the-shelf Controlnets is better than state-of-the-art AI portrait products in identity preservation and image fidelity. MegaPortrait has a simple but effective design and we compare it with other methods and products to show its superiority.
BibTeX
@misc{yang2024megaportrait,
  author = {Han Yang and Sotiris Anagnostidis and Enis Simsar and Thomas Hofmann},
  title = {{MegaPortrait: Revisiting Diffusion Control for High-fidelity Portrait Generation}},
  year = {2024},
  eprint = {2411.04357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04357},
}
Figure