UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment

Heming Zhu, Guoxing Sun, Christian Theobalt, Marc HabermannarXiv 2025 · DOI · arXiv

Introduction

Creating high-fidelity digital doubles is a cornerstone of immersive VR/AR and digital media. While current state-of-the-art models can generate convincing human avatars from a distance, they often lose photorealism upon closer inspection. UMA (Ultra-detailed Human Avatars) addresses this by introducing a framework capable of "yarn-level" detail, capturing the intricate textures and micro-folds of clothing that previous methods typically blur.

UMA achieves ultra-detailed geometry and texture rendering, capturing micro-structures like fabric weave.

The Challenge of Surface Misalignment

The primary obstacle to high-fidelity avatar creation is misalignment between the 3D model and the source video data. This typically manifests in two ways:

  1. Depth Misalignment: Skeletal motion is often insufficient to model complex, stochastic cloth dynamics, leading to a "one-to-many" mapping problem where a single pose corresponds to multiple possible surface shapes.
  2. Surface Drift: Even when the general volume is correct, the 3D "skin" may slide relative to the actual subject in the video.

When models attempt to learn textures from such misaligned data, the resulting "averaging" of errors produces a blurry, ghosted appearance. UMA aims to fix this tracking at the source to enable ultra-high-fidelity rendering.

Methodology: Multi-level Surface Alignment

UMA overcomes these issues through a three-stage pipeline designed to fix tracking at the source, moving from coarse global depth to fine individual texels.

The UMA pipeline: combining latent deformation, avatar-guided point tracking, and Gaussian texture super-resolution.

Latent Deformation Model

To handle non-rigid clothing wrinkles that skeletal poses cannot predict, UMA injects a per-frame learnable latent code. This allows the model to resolve ambiguities between pose and actual surface geometry, providing a more accurate base for texture mapping.

Avatar-Guided Point Tracking

Rather than relying solely on differentiable rendering—which is prone to local minima—UMA utilizes foundational 2D video point trackers. By projecting the 3D model into 2D, tracking points, and "lifting" them back into 3D, the system provides robust supervision for geometry at both the vertex and texel levels. This "trackers as teachers" approach is far more resilient to lighting changes than standard rendering-based losses.

Gaussian Texture Super-Resolution

The framework represents appearance using 3D Gaussian Splats attached to the mesh. To achieve micro-structure detail without prohibitive memory costs, a lightweight CNN-based module upsamples these Gaussians, effectively hallucinating fine details based on local context.

Evaluation and Results

The researchers introduced a new benchmark dataset featuring 5 subjects recorded with 40 calibrated 6K cameras to test UMA's capabilities. The subjects wore clothing with high-frequency patterns like floral prints and grids, performing diverse motions like jumping and dancing.

UMA significantly outperforms existing leaders such as ASH and TriHuman in both quantitative metrics (PSNR, LPIPS) and qualitative fidelity. While other methods produce "mushy" textures on checkered shirts, UMA maintains sharp lines and captures the physical "fuzz" of fabric. Remarkably, the model achieves this level of detail while rendering at 18 FPS (1620 × 3072 resolution), making it viable for high-end real-time applications.

Discussion and Future Work

UMA demonstrates that using off-the-shelf 2D trackers as a supervisor for 3D deformation is a powerful strategy for resolving surface drift. However, the current model is limited to a single-layered template, preventing applications like virtual try-on where one might swap clothing layers. Future work may also integrate external physics (e.g., wind response) and more nuanced facial capture to match the body's exceptional fidelity.

Summary updated

Abstract

Learning an animatable and clothed human avatar model with vivid dynamics and photorealistic appearance from multi-view videos is an important foundational research problem in computer graphics and vision. Fueled by recent advances in implicit representations, the quality of the animatable avatars has achieved an unprecedented level by attaching the implicit representation to drivable human template meshes. However, they usually fail to preserve the highest level of detail, particularly apparent when the virtual camera is zoomed in and when rendering at 4K resolution and higher. We argue that this limitation stems from inaccurate surface tracking, specifically, depth misalignment and surface drift between character geometry and the ground truth surface, which forces the detailed appearance model to compensate for geometric errors. To address this, we propose a latent deformation model and supervising the 3D deformation of the animatable character using guidance from foundational 2D video point trackers, which offer improved robustness to shading and surface variations, and are less prone to local minima than differentiable rendering. To mitigate the drift over time and lack of 3D awareness of 2D point trackers, we introduce a cascaded training strategy that generates consistent 3D point tracks by anchoring point tracks to the rendered avatar, which ultimately supervises our avatar at the vertex and texel level. To validate the effectiveness of our approach, we introduce a novel dataset comprising five multi-view video sequences, each over 10 minutes in duration, captured using 40 calibrated 6K-resolution cameras, featuring subjects dressed in clothing with challenging texture patterns and wrinkle deformations. Our approach demonstrates significantly improved performance in rendering quality and geometric accuracy over the prior state of the art.
BibTeX
@misc{zhu2025uma,
  author = {Heming Zhu and Guoxing Sun and Christian Theobalt and Marc Habermann},
  title = {{UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment}},
  year = {2025},
  eprint = {2506.01802},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.01802},
}
Figure