MI-NeRF: Learning a Single Face NeRF from Multiple Identities

Aggelina Chatziagapi, Grigorios G. Chrysos, Dimitris Samaras

Introduction

Neural Radiance Fields (NeRFs) have revolutionized the creation of photorealistic 3D digital humans. However, current state-of-the-art methods typically rely on per-identity optimization, requiring a dedicated model for every individual. This "one model per person" paradigm creates a significant scaling bottleneck: modeling 100 people requires 100 times the training resources. Furthermore, these specialist models often fail to generalize to expressions not present in a specific subject's training video. MI-NeRF (Multi-Identity NeRF) addresses these challenges by learning a single, unified 4D facial model capable of representing hundreds of identities simultaneously.

The Multiplicative Module

The core innovation of MI-NeRF is its architectural approach to disentangling identity from expression. Traditional multi-subject models often use simple concatenation of identity and expression vectors, which can lead to "identity leaking"—where the synthesized face appears as a blend of multiple people.

MI-NeRF introduces a Multiplicative Module ($M$) inspired by multilinear factor analysis. Instead of concatenation, the module computes a non-linear mapping based on the element-wise (Hadamard) product of identity and expression features. This provides a strong inductive bias, allowing the network to learn how specific expressions deform different facial structures. This module feeds into a unified dynamic NeRF backbone that predicts volume density and color based on 3D coordinates and viewing directions.

Experimental Setup and Datasets

To train a robust "generalist" model, the authors curated a diverse dataset of 140 high-definition talking face videos. This collection includes standard benchmarks like the Obama dataset, the PATS dataset (featuring presenters like Trevor Noah), and the LipNeRF dataset (featuring actors like Al Pacino).

MI-NeRF utilizes diverse datasets, including the Obama talking face videos, to learn shared facial dynamics across identities.

The model is conditioned on 3D Morphable Model (3DMM) parameters for expression, learned time-invariant identity codes, and per-frame latent codes to capture fine-grained details such as lighting changes. Of the 140 videos, 100 were used for training the base model, while the remainder served to test generalization to unseen identities.

Results and Efficiency

MI-NeRF demonstrates remarkable efficiency gains. Training a single model for 100 identities takes approximately 80 hours, a 90% reduction compared to the ~4,000 hours required to train 100 individual NeRF models.

Beyond speed, the model excels in expression transfer. Because it learns from a collective "dictionary" of facial movements, it can synthesize expressions for a target subject that were never seen in that subject's original video. For instance, a smile learned from one person can be accurately mapped onto another.

The model successfully synthesizes complex, high-energy expressions like shouting by leveraging knowledge shared across the entire multi-identity training set.

Quantitative evaluations show that MI-NeRF maintains identity better than single-identity baselines (like NeRFace) while producing fewer artifacts during extreme expression synthesis. Additionally, the model can be "personalized" to a completely new individual using as little as 1-3 seconds of video, significantly lowering the barrier for high-quality digital human creation.

Conclusion and Future Work

MI-NeRF represents a significant step toward scalable, high-fidelity digital humans. By moving from per-identity specialists to a unified generalist model, it achieves sublinear scaling and improved robustness. While currently dependent on 3DMM accuracy and limited in its handling of external factors like hair movement, the framework provides a foundation for future models trained on thousands of identities from massive "in-the-wild" video collections.

Summary updated

Abstract

In this work, we introduce a method that learns a single dynamic neural radiance field (NeRF) from monocular talking face videos of multiple identities. NeRFs have shown remarkable results in modeling the 4D dynamics and appearance of human faces. However, they require per-identity optimization. Although recent approaches have proposed techniques to reduce the training and rendering time, increasing the number of identities can be expensive. We introduce MI-NeRF (multi-identity NeRF), a single unified network that models complex non-rigid facial motion for multiple identities, using only monocular videos of arbitrary length. The core premise in our method is to learn the non-linear interactions between identity and non-identity specific information with a multiplicative module. By training on multiple videos simultaneously, MI-NeRF not only reduces the total training time compared to standard single-identity NeRFs, but also demonstrates robustness in synthesizing novel expressions for any input identity. We present results for both facial expression transfer and talking face video synthesis. Our method can be further personalized for a target identity given only a short video.
BibTeX
@misc{chatziagapi2024mi,
  author = {Aggelina Chatziagapi and Grigorios G. Chrysos and Dimitris Samaras},
  title = {{MI-NeRF: Learning a Single Face NeRF from Multiple Identities}},
  year = {2024},
  eprint = {2403.19920},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.19920},
}
Figure