Introduction
Generating hyper-realistic human faces from user-provided instructions—such as textual descriptions, sketches, or semantic masks—is a cornerstone of digital content creation. While multi-modal face synthesis has seen significant progress, existing frameworks often struggle with three critical dimensions: scalability (the ease of adding new control modalities), flexibility (the ability to combine arbitrary modalities at inference), and adaptivity (the capacity to balance different levels of information density across inputs).
Researchers from HKUST and Sun Yat-sen University have introduced a unified diffusion-based framework designed to address these hurdles. By leveraging Modal Surrogates and Entropy-Aware Modulation, the proposed system achieves state-of-the-art results in both image quality and alignment with complex, multi-modal user instructions.
Challenges in Multi-Modal Synthesis
Traditional face synthesis models typically face three major "pains":
- Poor Scalability: Adding a new control modality (e.g., lighting or 3D pose) often requires adding entirely new sub-networks or branches. This leads to bloated models that are computationally expensive to train and deploy.
- Rigidity: Many models are optimized for specific input pairings (e.g., text + mask). If a user provides a novel combination, such as sketch + lighting, the model may fail because it was not exposed to that specific configuration during training.
- Uniform Control Strength: Different modalities provide varying amounts of information. A semantic mask is highly precise (low entropy), whereas a text prompt like "a happy person" is vague (high entropy). Applying the same control strength to both often results in images that either ignore the text or over-fit to the mask, creating visual artifacts.
Methodology
The framework is built upon Latent Diffusion Models (LDM) and introduces two primary innovations to overcome the aforementioned challenges.
Modal Surrogates
Instead of dedicated sub-networks for every modality, the authors utilize Modal Surrogates—learnable tokens that act as "decorators" for the input conditions. These surrogates serve two functions:
- Condition Decoration: They help the network identify the specific characteristics of a given modality (e.g., distinguishing a sketch from a mask).
- Inter-modal Linkers: During training, surrogates for missing modalities are "invited" to participate. This allows the model to learn how different modalities should collaborate, enabling "zero-shot" combinations at inference time even when the model was trained primarily on uni-modal datasets.
Entropy-Aware Modal-Adaptive Modulation (EAM)
The EAM module acts as the system's "brain," dynamically deciding how much weight to give each input. Recognizing that high-entropy modalities (like text) require stronger guidance to influence the final image, while low-entropy modalities (like masks) require a lighter touch to avoid over-fitting, the EAM module adjusts diffusion noise levels at each step. This ensures that vague instructions are not drowned out by precise pixel-level maps.
Experimental Results
The model was trained on the CelebA-HQ dataset (30,000 high-quality images) using a uni-modal training strategy. This approach is highly efficient as it does not require a massive dataset where every image has a corresponding sketch, text, mask, and lighting map simultaneously.
Quantitative and Qualitative Performance
The method was compared against industry standards such as ControlNet, TediGAN, and T2I-Adapter.
- Quantitative: The proposed framework achieved a Fréchet Inception Distance (FID) of 103.14, significantly outperforming ControlNet (136.41) and T2I-Adapter (139.82) in multi-modal tasks.
- Qualitative: Generated faces demonstrate superior alignment with complex masks (such as specific hair shapes) while strictly adhering to textual attributes (like "no beard" or "arched eyebrows") that baseline models often ignored.
User Study
In a study involving 12 researchers, the proposed method consistently scored the highest in both image quality and condition alignment, validating the effectiveness of the adaptive modulation strategy.
Conclusion and Future Work
This research presents a significant step toward a truly flexible digital artist. By treating modality entropy as a control variable and using surrogates for scalability, the framework offers a robust solution for complex face synthesis. Future work may focus on reducing the computational cost inherent in diffusion models to enable real-time applications and exploring different initialization strategies for surrogates to further enhance performance.
Reference: Ren, J., et al. (2024). Towards Scalable, Flexible, and Adaptive Multi-Modal Face Synthesis. arXiv:2312.16274v2.
Summary updated