StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset

Chaofan Huo, Ye Shi, Yuexin Ma, Lan Xu, Jingyi Yu, Jingya Wang

Abstract

Reconstructing 3D human-object interactions (HOI) from a single RGB image is a fundamental challenge in computer vision, complicated by depth uncertainty and heavy occlusions. We propose StackFLOW, a novel framework that introduces the "Human-Object Offset" (HO-offset) representation to capture dense spatial relationships. By employing a two-stage Stacked Normalizing Flow model, we decouple the prediction of human pose and object-relative offsets. Our approach models these relationships as conditional probability distributions, providing a robust initialization for fast post-optimization. Experimental results on the BEHAVE and InterCap datasets demonstrate a 16% improvement in reconstruction accuracy and an 88% reduction in optimization time compared to previous state-of-the-art methods.

1. Introduction

Understanding how humans interact with their environment in 3D from 2D imagery is critical for applications in robotics, augmented reality, and human-computer interaction. Traditional methods often rely on sparse contact maps or implicit distance fields. However, these representations struggle with non-contact interactions and are computationally intensive to optimize.

Monocular reconstruction is inherently ill-posed; the same 2D projection can correspond to multiple 3D configurations. To address this, we need a representation that captures global spatial context and a model that can handle the probabilistic nature of the problem. StackFLOW addresses these needs by representing interactions through dense geometric offsets and modeling their distribution using generative flows.

2. Human-Object Offset Representation

A core contribution of this work is the Human-Object Offset (HO-offset). Unlike contact-based methods that only describe surface touching, we densely sample anchor points $\mathbf{p}^h_i$ on the human mesh and $\mathbf{p}^o_j$ on the object mesh. The offset vector is defined as $\mathbf{d}_{i,j} = \mathbf{p}^o_j - \mathbf{p}^h_i$.

The Human-Object Offset (HO-offset) captures dense geometric relationships between human and object surfaces.

To manage the high dimensionality of these dense offsets, we apply Principal Component Analysis (PCA) to project them into a low-dimensional latent spatial relation space. This captures the global correlation of the interaction, allowing the model to infer the position of occluded parts based on visible body segments.

3. The StackFLOW Framework

The StackFLOW architecture utilizes Normalizing Flows to map complex interaction distributions to a simple Gaussian space. The process is decoupled into two stages to ensure training stability and leverage the intuition that object placement is highly dependent on human posture:

  1. Human Pose Flow: Predicts the distribution of human poses $\theta$ conditioned on the input image features $c$.
  2. Offset Flow: Predicts the distribution of the HO-offset $\gamma$, conditioned on both the image features and the predicted human pose.
The StackFLOW pipeline: (a) Encoding spatial relations, (b) Posterior inference via stacked flows, and (c) Joint optimization for final reconstruction.

During inference, the model samples from these distributions to initialize a joint optimization process. This step minimizes 2D-3D reprojection errors while ensuring the final meshes adhere to the learned HO-offset constraints. Because the flow provides a highly accurate probabilistic starting point, the final refinement is significantly faster than traditional optimization-based methods.

4. Experimental Evaluation

We evaluated StackFLOW on the BEHAVE and InterCap datasets. To mitigate data scarcity, we employed free-viewport augmentation, generating 12 synthetic camera views for each training instance to simulate diverse occlusions.

4.1 Quantitative Results

StackFLOW significantly outperforms previous methods like CHORE. On the BEHAVE dataset, the method achieved a 16% reduction in Chamfer distance. More notably, the efficiency of the probabilistic initialization reduced the post-optimization time from 366 seconds to just 43 seconds per image—an 88% speedup.

4.2 Qualitative Performance

The model demonstrates remarkable robustness to heavy occlusions. By learning the global spatial relationship, StackFLOW can accurately place an object even when it is largely hidden behind the human subject, provided the human's pose is visible. This "global" geometric constraint is a major advantage over local contact-based priors.

5. Conclusion and Future Work

StackFLOW provides a powerful, efficient framework for monocular 3D HOI reconstruction. By combining a dense offset representation with stacked normalizing flows, we achieve state-of-the-art accuracy and speed. Current limitations include the assumption of pre-known object categories and bounding boxes. Future work will focus on integrating end-to-end object detection and extending the model to complex scenes with multiple interacting objects.

Summary updated

Abstract

Modeling and capturing the 3D spatial arrangement of the human and the object is the key to perceiving 3D human-object interaction from monocular images. In this work, we propose to use the Human-Object Offset between anchors which are densely sampled from the surface of human mesh and object mesh to represent human-object spatial relation. Compared with previous works which use contact map or implicit distance filed to encode 3D human-object spatial relations, our method is a simple and efficient way to encode the highly detailed spatial correlation between the human and object. Based on this representation, we propose Stacked Normalizing Flow (StackFLOW) to infer the posterior distribution of human-object spatial relations from the image. During the optimization stage, we finetune the human body pose and object 6D pose by maximizing the likelihood of samples based on this posterior distribution and minimizing the 2D-3D corresponding reprojection loss. Extensive experimental results show that our method achieves impressive results on two challenging benchmarks, BEHAVE and InterCap datasets.
BibTeX
@inproceedings{huo2023stackflow,
  author = {Chaofan Huo and Ye Shi and Yuexin Ma and Lan Xu and Jingyi Yu and Jingya Wang},
  title = {{StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/100},
}
Figure