Pose-Guided 3D Human Generation in Indoor Scene

Minseok Kim, Changwoo Kang, Jeongin Park, Kyungdon JooAAAI 2023 · DOI

Abstract

Generating realistic 3D human avatars that interact naturally with indoor environments is a complex task requiring precise scene-aware interaction. This paper presents a novel framework for generating 3D humans using a pose-guided geometric alignment strategy. By integrating a pose classifier within a Conditional Variational Autoencoder (cVAE), the model adaptively applies geometric constraints to relevant body parts. This approach ensures physically plausible contact—such as thighs touching a seat or a back resting on a bed—while significantly increasing pose diversity and eliminating the need for costly post-processing optimization.

Introduction

The placement of 3D human meshes in digital environments requires more than simple spatial positioning; it demands realistic physical interaction. A common failure in existing generation methods is the lack of "scene-awareness," where avatars appear to float above surfaces or interpenetrate (clip) through furniture. Traditional solutions often rely on global contact rules or heavy post-hoc optimization, which can stifle the variety of generated poses and increase computational overhead. The core challenge lies in developing a model that understands how different activities, such as sitting or lying down, necessitate distinct physical contact points with the environment.

Methodology

The proposed framework is built upon a Conditional Variational Autoencoder (cVAE) designed to generate human meshes conditioned on scene geometry. The innovation centers on two primary components:

  1. Pose Classifier: A compact MLP-based network that categorizes generated poses into four classes: standing, sitting, lying, or ambiguous.
  2. Adaptive Geometric Alignment Loss ($L_{geo}$): This loss function is applied selectively based on the pose classification. It consists of a point-wise distance metric (Chamfer distance) to pull specific body parts toward surfaces and a surface normal alignment constraint. The latter ensures that the orientation of the body mesh matches the contact surface, preventing clipping and ensuring a "flush" fit.
The proposed framework architecture featuring a cVAE, pose classifier, and geometric alignment loss.

By training the model to "snap" appropriate body segments to the environment, high-quality and physically plausible results are achieved directly during the inference stage.

Datasets

The research utilizes several key benchmarks:

  • PROX and PROX-E: Standard datasets for 3D human-scene interaction.
  • PROX-P (PROX-Pose): A new dataset introduced by the authors, featuring 12.5k manually annotated frames with pose labels and refined body segmentation (26 parts vs. the standard 8) for granular contact control.
  • MP3D-R: Used to evaluate the model's generalization capabilities in unseen room layouts.

Experimental Results

The method was evaluated against state-of-the-art models including PSI, PLACE, and POSA. Key findings include:

  • Physical Plausibility: The model achieved perfect scores (1.0) for both non-collision and contact metrics, indicating a complete absence of clipping and consistent environmental contact.
  • Pose Diversity: The variety of generated poses, measured by cluster size, reached 4.69—nearly double the 2.53 achieved by the PSI baseline.
  • Efficiency: Unlike previous methods that require "Geo Fitting" optimization steps, this framework produces high-fidelity results directly from the decoder.
Qualitative comparison of generated poses between the baseline PSI and the proposed method across different activities.

Conclusion

This work represents a significant advancement in scene-aware human generation by making contact constraints adaptive to the intended pose. While highly effective, limitations remain in handling extremely complex scene geometries and very fine-grained pose nuances. Future research will focus on expanding pose categories and extending the framework to dynamic motion sequences, moving beyond static snapshots to full interactive movements.

Summary updated

BibTeX
@inproceedings{kim2023pose,
  author = {Minseok Kim and Changwoo Kang and Jeongin Park and Kyungdon Joo},
  title = {{Pose-Guided 3D Human Generation in Indoor Scene}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25195},
}
Figure