Gaussian Control with Hierarchical Semantic Graphs in 3D Human Recovery

Hongsheng Wang, Weiyue Zhang, Sihao Liu, Xinrui Zhou, Shengyu Zhang, Fei Wu, Feng LinarXiv 2024 · DOI · OpenReview

Introduction

3D Gaussian Splatting (3DGS) has emerged as a powerful technique for real-time 3D rendering. However, its application to dynamic human recovery faces significant hurdles. Current state-of-the-art methods often suffer from "blurriness" at joint junctions (e.g., armpits, knees) and fail to preserve high-frequency surface details like clothing wrinkles or muscle definition. These issues stem from two primary factors: the neglect of inherent topological connectivity between body parts and the over-smoothing effect of standard 2D pixel-level supervision, which tends to average out complex 3D deformations.

The HUGS Framework

To address these challenges, we present HUGS (Human Gaussian Control with Hierarchical Semantic Graphs). This framework explicitly models kinematic relationships and surface details through two core modules: Inter-Semantic Kinematic Topology (SKT) and Intro-Surface Disentanglement (SD).

Inter-Semantic Kinematic Topology (SKT)

The SKT module focuses on global connectivity. By assigning semantic labels from the SMPL (Skinned Multi-Person Linear) model to each Gaussian, it constructs a 3D graph where edges represent spatial proximity. Using random walk algorithms for positional embeddings and a contrastive learning loss, the model learns the implicit linkage between adjacent body parts. This prevents "semantic confusion" and ensures sharp transitions at joints during motion.

Intro-Surface Disentanglement (SD)

The SD module targets local, high-frequency details. It clusters Gaussians by semantic labels and identifies "high-frequency nodes"—points exhibiting significant structural variance from their neighbors. The framework then selectively densifies Gaussians in these regions, allowing for the precise representation of sharp features like fabric folds and fine textures that are typically lost in standard optimization.

The HUGS architecture detailing the Inter-Semantic Kinematic Topology (SKT) and Intro-Surface Disentanglement (SD) modules.

Experimental Results

HUGS was evaluated on the ZJU-MoCap and MonoCap benchmarks, demonstrating superior performance in both visual fidelity and perceptual accuracy.

Quantitative Performance

On the ZJU-MoCap dataset, HUGS achieved state-of-the-art results, reaching a PSNR of 31.35 and an LPIPS (perceptual loss) of 28.93. These metrics significantly outperform recent strong baselines such as GauHuman and HumanNeRF, as shown in the performance comparison plot.

Qualitative Comparison

Visual results highlight the framework's ability to maintain crisp edges at body junctions and realistic clothing textures. Unlike previous models that "smear" details during movement, HUGS preserves the integrity of fine surface features like shirt folds and finger joints through its surgical, graph-based densification approach.

Comparison of semantic Gaussian representation, final output, and quantitative performance (PSNR vs LPIPS) against state-of-the-art methods.

Conclusion

By integrating hierarchical semantic graphs, HUGS provides a robust solution for high-fidelity 3D human recovery. The use of semantics as a structural guide prevents Gaussian bleeding and maintains structural integrity during fast or occluded motions. While the method currently relies on accurate initial pose estimation, future work will explore extending these graphs to model complex human-object interactions in diverse environments.

Summary updated

Abstract

Although 3D Gaussian Splatting (3DGS) has recently made progress in 3D human reconstruction, it primarily relies on 2D pixel-level supervision, overlooking the geometric complexity and topological relationships of different body parts. To address this gap, we introduce the Hierarchical Graph Human Gaussian Control (HUGS) framework for achieving high-fidelity 3D human reconstruction. Our approach involves leveraging explicitly semantic priors of body parts to ensure the consistency of geometric topology, thereby enabling the capture of the complex geometrical and topological associations among body parts. Additionally, we disentangle high-frequency features from global human features to refine surface details in body parts. Extensive experiments demonstrate that our method exhibits superior performance in human body reconstruction, particularly in enhancing surface details and accurately reconstructing body part junctions. Codes are available at https://wanghongsheng01.github.io/HUGS/.
BibTeX
@misc{wang2024gaussian,
  author = {Hongsheng Wang and Weiyue Zhang and Sihao Liu and Xinrui Zhou and Shengyu Zhang and Fei Wu and Feng Lin},
  title = {{Gaussian Control with Hierarchical Semantic Graphs in 3D Human Recovery}},
  year = {2024},
  eprint = {2405.12477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.12477},
}
Figure