StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset

Chaofan Huo, Ye Shi, Yuexin Ma, Lan Xu, Jingyi Yu, Jingya Wang

Abstract

Modeling and capturing the 3D spatial arrangement of the human and the object is the key to perceiving 3D human-object interaction (HOI) from monocular images. In this work, we propose to use the Human-Object Offset between anchors which are densely sampled from the surface of human mesh and object mesh to represent human-object spatial relation. Compared with previous works which use contact map or implicit distance fields to encode 3D human-object spatial relations, our method is a simple and efficient way to encode the highly detailed spatial correlation between the human and object. Based on this representation, we propose Stacked Normalizing Flow (StackFLOW) to infer the posterior distribution of human-object spatial relations from the image. During the optimization stage, we finetune the human body pose and object 6D pose by maximizing the likelihood of samples based on this posterior distribution and minimizing the 2D-3D corresponding reprojection loss. Extensive experimental results show that our method achieves impressive results on two challenging benchmarks, BEHAVE and InterCap datasets.

Introduction

Reconstructing 3D human-object interaction from a single RGB image is a fundamental challenge in computer vision due to inherent depth ambiguity and frequent occlusions. Traditional representations, such as contact maps, often fail to capture the full spatial context, especially in non-contact scenarios. Implicit distance fields, while more descriptive, suffer from high computational costs and slow optimization times.

To address these limitations, we introduce StackFLOW, a framework that rethinks HOI reconstruction through a dense, probabilistic representation. By modeling the spatial relationship as a set of offsets between surface anchors, we capture fine-grained interaction details in a computationally efficient manner, enabling both higher accuracy and significantly faster inference.

Methodology

The StackFLOW framework consists of three main components: spatial relation encoding, probabilistic inference, and joint optimization.

Human-Object Offset (HO-offset)

The core of our representation is the Human-Object Offset (HO-offset). We densely sample anchor points $p^h$ and $p^o$ from the human and object meshes, respectively. The spatial relation is defined by the set of 3D vectors $d_{i,j} = p_j^o - p_i^h$. To manage the high dimensionality of these offsets, we apply Principal Component Analysis (PCA) to learn a compact latent spatial relation space. This representation acts as a generalized contact map, providing informative signals even when the human and object are not in direct contact.

Stacked Normalizing Flow (StackFLOW)

To account for the uncertainty in monocular views, we employ Normalizing Flows to predict a posterior distribution of the spatial relations. We propose a "stacked" architecture that decouples the problem into two sequential steps:

  1. Human Pose Flow: Predicts the distribution of the human posture from image features.
  2. Offset Flow: Predicts the distribution of the HO-offset, conditioned on both the image and the predicted human pose.

This conditional structure mimics human perception, where understanding the person's action provides strong priors for the object's likely position.

Figure 1: The StackFLOW pipeline, illustrating (a) HO-offset encoding, (b) the stacked normalizing flow for posterior inference, and (c) the final post-optimization stage.

Post-optimization

The final reconstruction is refined through a joint optimization process. We maximize the likelihood of the predicted posterior distribution while minimizing the 2D-3D reprojection loss of human keypoints and object coordinate maps, ensuring the 3D model aligns perfectly with the input image.

Experiments and Results

We evaluated StackFLOW on the BEHAVE and InterCap datasets. Our method demonstrates significant improvements in both accuracy and efficiency over previous state-of-the-art methods like CHORE.

  • Accuracy: On the BEHAVE benchmark, StackFLOW achieved a 16% improvement in Chamfer distance compared to CHORE.
  • Efficiency: The optimization process is remarkably faster, requiring only 43 seconds per image compared to CHORE's 366 seconds—an 88% reduction in time.
  • Robustness: Qualitative results highlight the model's ability to reconstruct interactions accurately even under heavy occlusion, where competing methods often produce physically implausible poses or misaligned objects.
Figure 2: Qualitative comparison on the BEHAVE dataset. StackFLOW (Ours) maintains physical plausibility and accurate contact in scenarios where CHORE and BSTRO-HOI fail (highlighted in red).

Conclusion

StackFLOW introduces an efficient and robust approach to monocular 3D HOI reconstruction. By leveraging dense HO-offsets and a stacked probabilistic model, it overcomes the limitations of previous contact-based or implicit methods. Future work will explore extending this representation to multi-person interactions and dynamic, deformable objects.

Reference: Huo, C., Shi, Y., Ma, Y., Xu, L., Yu, J., & Wang, J. (2024). StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset. arXiv:2407.20545.

Summary updated

Abstract

Modeling and capturing the 3D spatial arrangement of the human and the object is the key to perceiving 3D human-object interaction from monocular images. In this work, we propose to use the Human-Object Offset between anchors which are densely sampled from the surface of human mesh and object mesh to represent human-object spatial relation. Compared with previous works which use contact map or implicit distance filed to encode 3D human-object spatial relations, our method is a simple and efficient way to encode the highly detailed spatial correlation between the human and object. Based on this representation, we propose Stacked Normalizing Flow (StackFLOW) to infer the posterior distribution of human-object spatial relations from the image. During the optimization stage, we finetune the human body pose and object 6D pose by maximizing the likelihood of samples based on this posterior distribution and minimizing the 2D-3D corresponding reprojection loss. Extensive experimental results show that our method achieves impressive results on two challenging benchmarks, BEHAVE and InterCap datasets.
BibTeX
@inproceedings{huo2023stackflow,
  author = {Chaofan Huo and Ye Shi and Yuexin Ma and Lan Xu and Jingyi Yu and Jingya Wang},
  title = {{StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/100},
}
Figure