Introduction
The creation of high-fidelity 3D digital avatars has seen significant advancement through Neural Radiance Fields (NeRF). However, most existing methods rely on static datasets where the subject's appearance remains constant. In practical applications, users frequently change clothing and adopt new poses over time. When a standard NeRF model is updated with new data sequentially, it typically suffers from "catastrophic forgetting"—a phenomenon where the model loses the ability to accurately render previously learned appearances.
MaintaAvatar introduces a framework designed for the continual learning of digital human avatars. It enables models to learn new appearances and poses from a limited number of images (few-shot learning) while maintaining the integrity of past "looks."
Challenges in Continual Avatar Learning
The authors identify two primary obstacles in the sequential training of human avatars:
- Color Bleeding: During the transition between learning tasks (e.g., changing from a red shirt to a blue jacket), textures from the previous task often interfere with the new one. This results in blurry artifacts or incorrect color representation as the model struggles to decouple task-specific features.
- Pose Forgetting: As the model adapts its deformation field to accommodate new body movements, it often loses the fine-tuned skeletal accuracy learned in earlier stages. This leads to distorted body shapes when attempting to animate the avatar using poses from previous datasets.
The MaintaAvatar Framework
To address these issues, MaintaAvatar builds upon the Deformable NeRF architecture, utilizing the SMPL (Skinned Multi-Person Linear) model to map various body poses back to a standardized "canonical" T-pose. The framework introduces three innovative components:
1. Global-Local Joint Storage Module
This module is the core solution to the color bleeding problem. Rather than relying on a single global code for appearance—which often causes interference—the authors employ a hybrid approach. They combine global embeddings (capturing general geometry and color) with Tri-planes (capturing local, fine-grained details). By utilizing Tri-planes for local storage, the model can spatially "pin" specific textures, drastically reducing the interference between different outfits.
2. Pose Distillation Module
To maintain pose accuracy across tasks, MaintaAvatar incorporates a "plug-and-play" Multi-Layer Perceptron (MLP) designed to correct errors in estimated body poses. During the continual learning process, the knowledge from the previous task's pose-corrector is "distilled" into the new version. This ensures that the avatar retains the ability to move correctly according to previously learned skeletal configurations.
3. Generative Replay Strategy
To avoid the prohibitive storage costs of keeping all previous training images, MaintaAvatar uses a generative replay strategy. A frozen version of the model from the preceding task generates "fake" supervision signals. These signals guide the current model, helping it remember the subject's appearance from earlier stages without requiring the original raw data.
Experimental Evaluation
The researchers evaluated MaintaAvatar using the ZJU-MoCap and THuman2.0 benchmarks. The model was tested in a challenging few-shot setting, utilizing as few as five images to learn a new task.
Quantitative Results
On the ZJU-MoCap dataset, MaintaAvatar achieved a PSNR of 29.495. This performance nearly matches the "Joint" baseline (29.507), which represents the theoretical upper limit where the model is trained on all data simultaneously. In contrast, existing continual learning methods like CLNeRF struggled significantly, achieving a PSNR of only approximately 12.3.
Qualitative Results
Visualizations demonstrate that MaintaAvatar maintains crisp textures and anatomically correct poses across multiple learning stages (Task 0 through Task 2). While competing models often exhibit "ghosting" effects or incorrect limb positions for past tasks, MaintaAvatar preserves high-fidelity rendering for all learned outfits.
Limitations and Future Directions
Despite its success, the framework faces certain limitations. The model struggles with extreme geometry changes, such as transitioning from tight-fitting clothes to very bulky attire, due to the inherent limits of the underlying deformation field. Additionally, because the model learns from a few-shot setup, it may struggle to generalize to extremely complex or unseen poses. Future research may focus on incorporating non-rigid clothing physics to enhance realism during high-intensity movements.
Conclusion
MaintaAvatar represents a significant step toward maintainable digital humans. By decoupling global and local features and implementing pose distillation, the framework effectively mitigates catastrophic forgetting and color bleeding, allowing for the seamless evolution of digital avatars in dynamic, real-world scenarios.
Summary updated