Abstract
Reconstructing 3D human-object interactions (HOI) from a single RGB image is a fundamental challenge in computer vision, complicated by depth uncertainty and heavy occlusions. We propose StackFLOW, a novel framework that introduces the "Human-Object Offset" (HO-offset) representation to capture dense spatial relationships. By employing a two-stage Stacked Normalizing Flow model, we decouple the prediction of human pose and object-relative offsets. Our approach models these relationships as conditional probability distributions, providing a robust initialization for fast post-optimization. Experimental results on the BEHAVE and InterCap datasets demonstrate a 16% improvement in reconstruction accuracy and an 88% reduction in optimization time compared to previous state-of-the-art methods.
1. Introduction
Understanding how humans interact with their environment in 3D from 2D imagery is critical for applications in robotics, augmented reality, and human-computer interaction. Traditional methods often rely on sparse contact maps or implicit distance fields. However, these representations struggle with non-contact interactions and are computationally intensive to optimize.
Monocular reconstruction is inherently ill-posed; the same 2D projection can correspond to multiple 3D configurations. To address this, we need a representation that captures global spatial context and a model that can handle the probabilistic nature of the problem. StackFLOW addresses these needs by representing interactions through dense geometric offsets and modeling their distribution using generative flows.
2. Human-Object Offset Representation
A core contribution of this work is the Human-Object Offset (HO-offset). Unlike contact-based methods that only describe surface touching, we densely sample anchor points $\mathbf{p}^h_i$ on the human mesh and $\mathbf{p}^o_j$ on the object mesh. The offset vector is defined as $\mathbf{d}_{i,j} = \mathbf{p}^o_j - \mathbf{p}^h_i$.
To manage the high dimensionality of these dense offsets, we apply Principal Component Analysis (PCA) to project them into a low-dimensional latent spatial relation space. This captures the global correlation of the interaction, allowing the model to infer the position of occluded parts based on visible body segments.
3. The StackFLOW Framework
The StackFLOW architecture utilizes Normalizing Flows to map complex interaction distributions to a simple Gaussian space. The process is decoupled into two stages to ensure training stability and leverage the intuition that object placement is highly dependent on human posture:
- Human Pose Flow: Predicts the distribution of human poses $\theta$ conditioned on the input image features $c$.
- Offset Flow: Predicts the distribution of the HO-offset $\gamma$, conditioned on both the image features and the predicted human pose.
During inference, the model samples from these distributions to initialize a joint optimization process. This step minimizes 2D-3D reprojection errors while ensuring the final meshes adhere to the learned HO-offset constraints. Because the flow provides a highly accurate probabilistic starting point, the final refinement is significantly faster than traditional optimization-based methods.
4. Experimental Evaluation
We evaluated StackFLOW on the BEHAVE and InterCap datasets. To mitigate data scarcity, we employed free-viewport augmentation, generating 12 synthetic camera views for each training instance to simulate diverse occlusions.
4.1 Quantitative Results
StackFLOW significantly outperforms previous methods like CHORE. On the BEHAVE dataset, the method achieved a 16% reduction in Chamfer distance. More notably, the efficiency of the probabilistic initialization reduced the post-optimization time from 366 seconds to just 43 seconds per image—an 88% speedup.
4.2 Qualitative Performance
The model demonstrates remarkable robustness to heavy occlusions. By learning the global spatial relationship, StackFLOW can accurately place an object even when it is largely hidden behind the human subject, provided the human's pose is visible. This "global" geometric constraint is a major advantage over local contact-based priors.
5. Conclusion and Future Work
StackFLOW provides a powerful, efficient framework for monocular 3D HOI reconstruction. By combining a dense offset representation with stacked normalizing flows, we achieve state-of-the-art accuracy and speed. Current limitations include the assumption of pre-known object categories and bounding boxes. Future work will focus on integrating end-to-end object detection and extending the model to complex scenes with multiple interacting objects.
Summary updated