RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians

Sen Peng, Weixing Xie, Zilong Wang, Xiaohu Guo, Zhonggui Chen, Baorong Yang, Xiao Dong

Abstract

Creating high-fidelity, animatable human avatars from a single-view (monocular) video remains a significant challenge in computer vision. Current methods often struggle to balance the efficiency of geometric priors with the complex, non-rigid deformations of clothing and hair. We present RMAvatar, a novel hybrid representation that embeds 3D Gaussian Splats onto a deformable mesh. By introducing a "rectification" module, RMAvatar effectively compensates for the limitations of traditional Linear Blend Skinning (LBS), allowing for the capture of fine-grained, pose-dependent details. Experimental results on standard benchmarks demonstrate that RMAvatar achieves state-of-the-art rendering quality and superior generalization to unseen poses compared to existing NeRF-based and point-based approaches.

Introduction

The synthesis of photorealistic digital humans from monocular video is a cornerstone technology for applications in telepresence, gaming, and virtual reality. However, this task is inherently under-constrained, as a single camera view cannot capture the full geometry of a subject, leading to potential "hallucination" errors. Furthermore, traditional animation techniques like Linear Blend Skinning (LBS) are often too rigid to represent the fluid dynamics of loose clothing or skin folds. While Neural Radiance Fields (NeRF) have improved visual quality, they are often computationally expensive and prone to producing blurry results.

RMAvatar addresses these hurdles by combining the structural benefits of a mesh template with the high-frequency detail of 3D Gaussian Splatting (3DGS). By embedding Gaussians directly into the local coordinate systems of a mesh and applying a learned rectification module, the system can represent complex non-rigid motions that traditional skinning methods fail to capture.

Methodology

RMAvatar utilizes a two-stage pipeline designed to bridge the gap between rigid skeletal movement and fluid surface appearance.

1. Mesh-Embedded Gaussian Initialization

The process begins with a standard SMPL (Skinned Multi-Person Linear) human body template. 3D Gaussians are initialized and "bound" to the local coordinate systems of individual triangular faces on the mesh. As the mesh deforms according to the subject's pose via LBS, the Gaussians follow the underlying geometry. This provides a strong geometric prior that ensures the avatar remains topologically consistent and free of structural holes.

2. Gaussian Rectification Module

The core innovation of RMAvatar is the Gaussian Rectification Module. Recognizing that LBS cannot account for the independent movement of fabric or hair, the authors designed a pose-dependent Multi-Layer Perceptron (MLP). This module predicts specific offsets for the Gaussians—including position ($\delta u$), rotation ($\delta r$), and scale ($\delta s$)—based on the current pose. This allows the Gaussians to "drift" or adjust relative to their parent triangles, effectively "rectifying" the errors of the underlying mesh deformation and capturing dynamic details like bunching fabric or shifting wrinkles.

3. Optimization and Density Control

To ensure high-fidelity reconstruction in complex areas such as the face or intricate clothing, RMAvatar employs Adaptive Density Control. The system adaptively splits or clones Gaussians in high-detail regions while pruning those that do not contribute to the final image. To maintain physical plausibility, the authors implement regularization losses ($\mathcal{L}_{offset}$ and $\mathcal{L}_{scaling}$) that prevent Gaussians from deviating too far from the mesh surface or becoming unnaturally elongated.

Experimental Evaluation

Datasets

The model was evaluated on two prominent benchmarks:

  • PeopleSnapshot: Features subjects performing self-rotation in casual attire, providing a baseline for standard monocular reconstruction.
  • ZJU-MoCap: Includes complex, large-scale human motions. Although this dataset provides multi-view data, RMAvatar was trained using only monocular input to test its performance under constrained conditions.

Results and Comparative Analysis

RMAvatar demonstrates a clear quantitative and qualitative lead over existing state-of-the-art (SOTA) methods such as GaussianAvatar and SplattingAvatar.

  • Quantitative Performance: On the PeopleSnapshot dataset, RMAvatar achieved a PSNR of 34.12, a significant improvement over SplattingAvatar’s 32.31. This indicates a much higher level of reconstruction accuracy.
  • Visual Fidelity: Qualitatively, RMAvatar produces sharper facial features and more realistic clothing textures. While previous methods often suffer from "burr" artifacts or blurriness on loose garments, the rectification module in RMAvatar successfully mitigates these issues.
  • Generalization: When tested on "out-of-distribution" poses (poses not encountered during training), RMAvatar maintained consistent 3D shapes and smooth boundaries, showcasing its robustness for animation.

Discussion and Limitations

While RMAvatar represents a significant step forward, it is not without limitations. The model remains dependent on the SMPL template; consequently, it may struggle with extremely bulky clothing (e.g., heavy parkas) or very long hair that deviates significantly from the standard human body shape. Additionally, while the model handles body poses expertly, its ability to capture fine-grained facial expressions or intricate finger movements is currently limited.

Future work may involve the integration of layered meshes—separating the body from the clothing—combined with Gaussian Splatting to further enhance the reconstruction of complex, multi-layered outfits.

Conclusion

RMAvatar introduces a powerful hybrid approach to human avatar creation. By embedding Gaussians into a mesh and applying a learned rectification layer, it overcomes the rigidity of traditional skinning while maintaining the efficiency and detail of 3D Gaussian Splatting. The result is a photorealistic, animatable avatar capable of high-fidelity rendering from simple monocular video sources.

Summary updated

BibTeX
@article{peng2025rmavatar,
  author = {Sen Peng and Weixing Xie and Zilong Wang and Xiaohu Guo and Zhonggui Chen and Baorong Yang and Xiao Dong},
  title = {{RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians}},
  journal = {Graph. Model.},
  year = {2025},
  doi = {10.1016/j.gmod.2025.101266},
}
Figure