Towards Flexible, Scalable, and Adaptive Multi-Modal Conditioned Face Synthesis

Jingjing Ren, Cheng Xu, Haoyu Chen, Xinran Qin, Chongyi Li, Lei ZhuarXiv 2023 · DOI · arXiv

Introduction

Generating hyper-realistic human faces from user-provided instructions—such as textual descriptions, sketches, or semantic masks—is a cornerstone of digital content creation. While multi-modal face synthesis has seen significant progress, existing frameworks often struggle with three critical dimensions: scalability (the ease of adding new control modalities), flexibility (the ability to combine arbitrary modalities at inference), and adaptivity (the capacity to balance different levels of information density across inputs).

Researchers from HKUST and Sun Yat-sen University have introduced a unified diffusion-based framework designed to address these hurdles. By leveraging Modal Surrogates and Entropy-Aware Modulation, the proposed system achieves state-of-the-art results in both image quality and alignment with complex, multi-modal user instructions.

Challenges in Multi-Modal Synthesis

Traditional face synthesis models typically face three major "pains":

  1. Poor Scalability: Adding a new control modality (e.g., lighting or 3D pose) often requires adding entirely new sub-networks or branches. This leads to bloated models that are computationally expensive to train and deploy.
  2. Rigidity: Many models are optimized for specific input pairings (e.g., text + mask). If a user provides a novel combination, such as sketch + lighting, the model may fail because it was not exposed to that specific configuration during training.
  3. Uniform Control Strength: Different modalities provide varying amounts of information. A semantic mask is highly precise (low entropy), whereas a text prompt like "a happy person" is vague (high entropy). Applying the same control strength to both often results in images that either ignore the text or over-fit to the mask, creating visual artifacts.

Methodology

The framework is built upon Latent Diffusion Models (LDM) and introduces two primary innovations to overcome the aforementioned challenges.

Instead of dedicated sub-networks for every modality, the authors utilize Modal Surrogates—learnable tokens that act as "decorators" for the input conditions. These surrogates serve two functions:

  • Condition Decoration: They help the network identify the specific characteristics of a given modality (e.g., distinguishing a sketch from a mask).
  • Inter-modal Linkers: During training, surrogates for missing modalities are "invited" to participate. This allows the model to learn how different modalities should collaborate, enabling "zero-shot" combinations at inference time even when the model was trained primarily on uni-modal datasets.

Entropy-Aware Modal-Adaptive Modulation (EAM)

The EAM module acts as the system's "brain," dynamically deciding how much weight to give each input. Recognizing that high-entropy modalities (like text) require stronger guidance to influence the final image, while low-entropy modalities (like masks) require a lighter touch to avoid over-fitting, the EAM module adjusts diffusion noise levels at each step. This ensures that vague instructions are not drowned out by precise pixel-level maps.

Proposed architecture with Modal Surrogates and Control Strength Adjust Layer.

Experimental Results

The model was trained on the CelebA-HQ dataset (30,000 high-quality images) using a uni-modal training strategy. This approach is highly efficient as it does not require a massive dataset where every image has a corresponding sketch, text, mask, and lighting map simultaneously.

Quantitative and Qualitative Performance

The method was compared against industry standards such as ControlNet, TediGAN, and T2I-Adapter.

  • Quantitative: The proposed framework achieved a Fréchet Inception Distance (FID) of 103.14, significantly outperforming ControlNet (136.41) and T2I-Adapter (139.82) in multi-modal tasks.
  • Qualitative: Generated faces demonstrate superior alignment with complex masks (such as specific hair shapes) while strictly adhering to textual attributes (like "no beard" or "arched eyebrows") that baseline models often ignored.
Qualitative results showing synthesis across various modalities and their combinations.

User Study

In a study involving 12 researchers, the proposed method consistently scored the highest in both image quality and condition alignment, validating the effectiveness of the adaptive modulation strategy.

Conclusion and Future Work

This research presents a significant step toward a truly flexible digital artist. By treating modality entropy as a control variable and using surrogates for scalability, the framework offers a robust solution for complex face synthesis. Future work may focus on reducing the computational cost inherent in diffusion models to enable real-time applications and exploring different initialization strategies for surrogates to further enhance performance.

Reference: Ren, J., et al. (2024). Towards Scalable, Flexible, and Adaptive Multi-Modal Face Synthesis. arXiv:2312.16274v2.

Summary updated

Abstract

Recent progress in multi-modal conditioned face synthesis has enabled the creation of visually striking and accurately aligned facial images. Yet, current methods still face issues with scalability, limited flexibility, and a one-size-fits-all approach to control strength, not accounting for the differing levels of conditional entropy, a measure of unpredictability in data given some condition, across modalities. To address these challenges, we introduce a novel uni-modal training approach with modal surrogates, coupled with an entropy-aware modal-adaptive modulation, to support flexible, scalable, and scalable multi-modal conditioned face synthesis network. Our uni-modal training with modal surrogate that only leverage uni-modal data, use modal surrogate to decorate condition with modal-specific characteristic and serve as linker for inter-modal collaboration , fully learns each modality control in face synthesis process as well as inter-modal collaboration. The entropy-aware modal-adaptive modulation finely adjust diffusion noise according to modal-specific characteristics and given conditions, enabling well-informed step along denoising trajectory and ultimately leading to synthesis results of high fidelity and quality. Our framework improves multi-modal face synthesis under various conditions, surpassing current methods in image quality and fidelity, as demonstrated by our thorough experimental results.
BibTeX
@misc{ren2023towards,
  author = {Jingjing Ren and Cheng Xu and Haoyu Chen and Xinran Qin and Chongyi Li and Lei Zhu},
  title = {{Towards Flexible, Scalable, and Adaptive Multi-Modal Conditioned Face Synthesis}},
  year = {2023},
  eprint = {2312.16274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.16274},
}
Figure