Introduction
Personalized portrait generation has long struggled with the "identity-style dilemma"—a fundamental trade-off between maintaining the recognizable features of a subject (identity preservation) and accurately adopting the aesthetic qualities of a target environment (artistic adaptation). Existing diffusion-based models often fail by either washing out the subject's identity to match the style or producing "pasted-on" faces that clash with the background's lighting and geometry. MegaPortrait addresses this by decoupling facial geometry from shading through a novel "split-and-merge" pipeline.
Methodology: The Split-and-Merge Pipeline
MegaPortrait is built upon Stable Diffusion V1.5 and ControlNet, utilizing a modular architecture to handle different aspects of the portrait generation process.
Decoupling via Frequency
The core innovation lies in the realization that identity is primarily encoded in high-frequency (HF) geometric details, while style and lighting are represented in low-frequency (LF) shading information. The system employs three specialized networks:
- Identity Net: Fine-tuned via LoRA on a small set of source images, this module generates HF-Maps (capturing geometry) and Light Maps (capturing shading).
- Shading Net: This network uses the HF-Maps and Light Maps as control signals to synthesize a face that maintains the source identity but adopts the reference scene's lighting.
- Harmonization Net: This final stage ensures a seamless transition between the generated face and the reference image's body and background, correcting color grading and edge artifacts.
Gaze Correction
To solve the "dead eyes" problem common in overfitted personalized models, MegaPortrait introduces a "free-lunch" gaze correction technique. By applying a local mask over the eyes and using classifier-free guidance with text prompts like "looking at the camera," the model restores expressive, vivid gazes without requiring additional training or complex architectural changes.
Experimental Results
The model was evaluated on a curated benchmark of 100 unique identity-style pairs. Quantitative analysis using ArcFace (for identity similarity) and the Aesthetic Score Predictor V2 shows that MegaPortrait outperforms leading tools like Remini and IP-Adapter.
| Model | Aesthetic Score | ID Score |
|---|---|---|
| MegaPortrait | 6.3789 | 0.6630 |
| Remini | 6.2431 | 0.6157 |
| IP-Adapter-FaceID | 6.1062 | 0.6580 |
Qualitatively, MegaPortrait excels in preserving the "soul" of the identity even under extreme lighting or non-photorealistic artistic styles, such as oil paintings or dramatic noir photography.
Limitations and Future Directions
While MegaPortrait sets a new standard for high-fidelity portraits, some challenges remain. The model occasionally inherits hair or eye colors from the reference style rather than the source identity. Furthermore, the authors suggest that future iterations could benefit from "aesthetic mining"—extracting user preferences from edited portraits—and exploring non-spatial ControlNets to better decouple facial expressions from identity.
Conclusion
By revisiting diffusion control through the lens of frequency decoupling, MegaPortrait effectively resolves the identity-style conflict. Its modular approach allows for high-fidelity, personalized generation that respects both the subject's unique features and the artistic integrity of the reference environment.
Summary updated