Real-time High-fidelity Gaussian Human Avatars with Position-based Interpolation of Spatially Distributed MLPs

Youyi Zhan, Tianjia Shao, Yin Yang, Kun Zhou

TL;DR

This paper introduces a novel method for creating high-fidelity, real-time animated 3D human avatars. By distributing neural networks across the avatar's body and using a clever interpolation scheme for appearance details, they achieve significantly faster rendering speeds while preserving fine-grained, pose-dependent visual features, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Creating realistic 3D human avatars that can be animated in real-time is a significant challenge. Existing methods often fall into two camps: either they use a single neural network to capture appearance, which struggles with fine details and pose-dependent changes, or they employ computationally intensive networks that produce high-quality results but at a slow, non-real-time rendering speed (around 10 FPS). This leaves a gap for methods that can achieve both high fidelity and real-time performance, especially when animating avatars under novel poses and from new viewpoints.

Methods

The core innovation lies in the Spatially Distributed MLPs approach. Instead of a single, large neural network, the authors distribute multiple smaller MLPs across different "anchor points" on a template human mesh. Each MLP is responsible for learning local appearance changes based on the human's pose.

Key aspects of their method include:

  • Position-based Interpolation: Each Gaussian (a fundamental element in 3D Gaussian Splatting) gets its appearance parameters by interpolating from the outputs of its nearest MLPs. This reduces the computational load compared to querying a single large network for every Gaussian.
  • Gaussian Offset Basis: To avoid overly smooth or "blurry" appearance changes during interpolation, they introduce a system of "Gaussian offset basis." The MLPs predict coefficients for these bases, allowing for distinct property offsets for each Gaussian while the coefficients themselves vary smoothly. This enables the capture of high-frequency spatial details.
  • Control Points for Surface Constraint: To prevent Gaussians from moving irregularly inside the avatar's body during animation (which can cause artifacts), they use "control points" on the surface of the template mesh. The position offsets of Gaussians are interpolated from these control points, ensuring they remain on the surface and animate more realistically.
  • vmap for Parallel Processing: They leverage PyTorch's vmap function for efficient parallel processing of the spatially distributed MLPs, significantly speeding up computation.

Datasets

The authors evaluate their method on several public datasets:

  • AvatarRex [59]: Used for experiments with full-body videos from 16 views.
  • THuman4.0 [58]: Features videos of people with rich appearance details, captured at 1K resolution.
  • ActorsHQ [15]: A high-quality dataset used with 39 full-body views.

They also mention using SMPL-X [39] registrations for pose information.

Key Insights and Ideas

  • Decentralizing Neural Networks: Distributing MLPs across the body is more effective for capturing local, pose-dependent details than a single monolithic network. This reduces the burden on each individual network.
  • Basis-Coefficient Duality: The separation of learned "basis" (representing fundamental appearance changes) and interpolated "coefficients" is crucial. It allows for smooth interpolation of coefficients while still enabling the representation of sharp, high-frequency details through the basis.
  • Surface Constraint is Key for Animation: Simply interpolating position offsets can lead to Gaussians moving inside the body, causing artifacts. Using control points to enforce surface-based animation is vital for realistic motion.
  • vmap is a Game-Changer for Performance: The efficient parallelization of MLPs using vmap is a significant factor in achieving real-time rendering speeds.

Key Results

  • Rendering Speed: Achieved a rendering speed of 166 FPS under novel views and novel poses, a dramatic improvement over state-of-the-art methods like AnimatableGaussians (10 FPS).
  • Appearance Quality: Outperforms methods like 3DGS-Avatar and MeshAvatar in capturing fine wrinkles and textures. Achieves comparable or better results than AnimatableGaussians, with finer details.
  • Quantitative Metrics: Achieved the best PSNR, SSIM, LPIPS, and FID scores compared to other methods under both training and novel poses (e.g., PSNR of 32.7456, SSIM of 0.9868, LPIPS of 0.0226, FID of 10.1169).
  • Training Efficiency: Significantly reduced training time compared to AnimatableGaussians (17.5 hours vs. 100 hours).

Limitations and Future Work

  • Appearance Conditioning: The current avatar appearance is solely conditioned on pose. It cannot model complex cloth dynamics like swaying skirts.
  • Heavy Pipeline: The method still relies on multi-view capture, pose estimation, and template mesh extraction, making the overall pipeline quite heavy.
  • Future Directions:
    • Modeling clothes as a separate layer and incorporating simulation for more complex dynamics.
  • Exploring reconstruction from monocular videos for greater accessibility.
  • Reducing pipeline complexity by using fewer RGB-D cameras, which could simplify skeleton estimation and mesh reconstruction.

Summary updated

BibTeX
@inproceedings{zhan2025real,
  author = {Youyi Zhan and Tianjia Shao and Yin Yang and Kun Zhou},
  title = {{Real-time High-fidelity Gaussian Human Avatars with Position-based Interpolation of Spatially Distributed MLPs}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02449},
}
Figure