Semantic-Human: Neural Rendering of Humans from Monocular Video with Human Parsing

Jie Zhang, Pengcheng Shi, Zaiwang Gu, Yiyang Zhou, Zhi WangarXiv 2023 · DOI · arXiv

Abstract

Semantic-Human is a novel neural rendering framework that jointly learns a person's appearance, 3D geometry, and semantic body parts (parsing) from a single monocular video. By integrating human parsing into the Neural Radiance Field (NeRF) pipeline, the model produces photorealistic novel views while simultaneously generating viewpoint-consistent semantic labels. This approach effectively "denoises" inconsistent 2D labels into a coherent 3D representation. Experimental results on the ZJU-MoCap dataset demonstrate that Semantic-Human achieves competitive rendering performance while providing high-quality human parsing, enabling applications such as label synthesis and consistent image editing.

Introduction

Creating high-quality 3D digital humans from a single camera (monocular video) is a major challenge in computer vision. While existing methods like HumanNeRF can produce impressive visual results, they are "semantically blind"—they process pixels without understanding which belong to specific body parts like a "left arm" or "torso."

Standard 2D human parsing tools exist but operate frame-by-frame, often leading to "flickering" results that lack temporal and viewpoint consistency. Furthermore, mapping a moving person back to a static "canonical" 3D space to learn their shape often results in blurry edges when motion is complex. Semantic-Human bridges this gap by creating a model that understands both the appearance and the logic of the human body.

Methods

Semantic-Human builds upon HumanNeRF, using a motion field to map points from an observed frame back to a standardized "T-pose" canonical space. The framework introduces three key innovations:

  1. Joint Semantic-Radiance Fields: The standard NeRF MLP is extended with an additional segmentation head, allowing the network to output color, density, and a semantic label for every 3D point.
  2. Surface and Silhouette Constraints:
  3. Surface Loss: Uses the SMPL 3D human body model as a geometric prior to ensure motion field accuracy.
  4. Silhouette Loss: Incorporates binary masks to regularize volumetric geometry, leading to sharper boundaries.
  5. Delayed Optimization: To prevent noisy initial 2D semantic labels from degrading geometry, the human parsing loss is introduced only after 50,000 iterations, once basic shape and appearance have stabilized.
The Semantic-Human architecture integrates motion fields with joint appearance and semantic rendering heads.

Key Results and Insights

The model was evaluated on the ZJU-MoCap dataset, featuring subjects performing complex motions. Quantitatively, Semantic-Human achieved a PSNR of 30.42 and an SSIM of 0.9769 for Subject 377, slightly outperforming the HumanNeRF baseline while adding semantic parsing capabilities.

NeRF as a Denoiser

A key insight is that NeRF's inherent 3D consistency acts as a natural filter. By representing semantics in a 3D canonical space, noisy 2D labels from off-the-shelf segmenters are smoothed into a clean, viewpoint-consistent field. This joint learning also improves geometry, as semantic boundaries help define sharper geometric edges.

Qualitative Performance

Visually, the model produces cleaner boundaries and fewer artifacts compared to previous methods. It enables several downstream applications:

  • Label Synthesis and Denoising: Generating smooth semantic maps for new viewpoints.
  • Consistent Image Editing: Allowing users to change the color of specific body parts across a video sequence with perfect consistency.
Comparison of novel view synthesis shows that Semantic-Human (Ours) produces more detailed and consistent results than HumanNeRF.

Conclusion

Semantic-Human demonstrates that appearance, geometry, and semantics are intrinsically linked. By learning them jointly, the model achieves superior rendering and consistent human parsing from monocular video. Future work will focus on increasing label granularity and improving performance under extremely sparse supervision.

Summary updated

BibTeX
@misc{zhang2023semantic,
  author = {Jie Zhang and Pengcheng Shi and Zaiwang Gu and Yiyang Zhou and Zhi Wang},
  title = {{Semantic-Human: Neural Rendering of Humans from Monocular Video with Human Parsing}},
  year = {2023},
  eprint = {2308.09894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.09894},
}
Figure