UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video

Tanuj Sur, Shashank Tripathi, Nikos Athanasiou, Ha Linh Nguyen, Kai Xu, Michael J. Black, Angela YaoarXiv 2026 · arXiv

TL;DR

UniCon3R is a unified feed-forward framework that achieves physically plausible 4D human-scene reconstruction from a single monocular video stream. By treating human-scene contact as an internal corrective feedback loop rather than a passive auxiliary prediction, it significantly eliminates artifacts like floating or floor penetration while maintaining fast, online streaming speeds.

What Problem Does This Paper Address?

Simultaneously reconstructing a 3D scene and tracking human motion from a monocular video is a long-standing challenge in computer vision. While recent feed-forward models can estimate scene geometry and human meshes in a single pass, they lack physical awareness. As a result, reconstructed human bodies frequently suffer from glaring artifacts: feet floating in mid-air or legs clipping straight through the floor.

The root cause is that existing models process the scene geometry and human mesh recovery branches largely in isolation. They fail to explicitly model how the human interacts with the environment, missing out on the powerful physical constraints that contact provides.

Figure 1: Contact improves unified 4D human-scene reconstruction. (a) Human3R [7] meshes are misaligned, with the feet penetrating the ground (dashed yellow circles). UniCon3R predicts dense per-vertex contact (green inset) and uses it to improve body-scene alignment. (b) Amortized runtime versus global human motion error . UniCon3R achieves the state-of-the-art on the RICH dataset [14]; test-time optimization further reduces the error, but also reduces runtime efficiency.

Methods

UniCon3R introduces a contact-aware architecture that integrates physical interaction directly into the network's forward pass. Built on top of frozen scene-reconstruction (CUT3R) and human-prompting (Human3R) backbones, the framework introduces two novel components:

  1. Contact Prompt Fusion (§ 4.1): The model aggregates interaction evidence from three distinct sources to form a scene-aware contact prompt:
  2. Semantic Scene Context: Combines immediate visual cues from the current frame with persistent object memory from previous frames using cross-attention.
  3. Explicit Metric Geometry: Injects local 3D support cues by pooling the previous frame's world pointmap around the human anchor using RoIAlign.
  4. Temporal Momentum: Reuses and projects the contact token from the previous frame to stabilize predictions against occlusions.
  5. Contact-Guided Latent Refinement (§ 4.2): Instead of treating contact as a final, isolated output ("Parallel Readout"), UniCon3R feeds the refined contact token back into the human mesh recovery branch. It generates a residual update ($\Delta H_t$) that actively corrects the human latent representation before regressing the final SMPL-X body pose parameters.
Overview of the UniCon3R architecture and its Contact Prompt Fusion module

Datasets

The model is fine-tuned on a balanced mixture of two datasets:

  • RICH: A real-world dataset containing multi-camera video, scanned 3D scene geometry, and dense full-body contact annotations.
  • BEDLAM: A large-scale synthetic dataset of diverse human bodies and lifelike animations.

For evaluation, the authors test global human motion estimation and physical plausibility on RICH and EMDB-2, and local human mesh recovery on SLOPER4D and 3DPW.

Key Insights and Ideas

  • Passive Prediction is Insufficient: Simply supervising a network to predict contact as a side-task (Parallel Readout) does not improve the final pose. To fix physical implausibility, the contact information must actively reshape the latent features guiding the pose generator.
  • Bidirectional Synergy: Internalizing contact creates a win-win scenario. Not only does it make the human pose physically grounded, but the joint optimization also boosts the accuracy and recall of the contact predictions themselves.
  • Structured Attention Routing: Layerwise analysis reveals that the network naturally learns a structured pipeline: human tokens guide contact reasoning in early decoder layers, while contact tokens provide corrective feedback to human tokens in later layers.

Key Results

  • Enhanced Physical Plausibility: On the RICH dataset, UniCon3R reduces floor penetration by 74% and body floating by 83% compared to the contact-agnostic Human3R baseline.
  • Improved Motion Tracking: It improves the global human motion metric (WA-MPJPE) by 16% on RICH, outperforming all state-of-the-art feed-forward baselines.
  • Negligible Computational Overhead: Despite the added contact reasoning, UniCon3R runs at 2.40 FPS (using a ViT-L backbone), which is virtually identical to the baseline's speed (2.41 FPS). A lightweight variant (ViT-S) achieves 5.64 FPS while preserving most of the accuracy gains, offering a clean trade-off for compute-constrained applications.

Limitations and Future Work

  • Simplistic Contact Representation: Contact is currently modeled as binary per-vertex labels. It does not account for continuous physical forces, proximity, or friction.
  • Rigid Scene Assumption: The framework assumes the environment consists of rigid support geometry, meaning it cannot accurately model interactions with deformable objects (like sitting on a soft cushion).
  • Error Propagation: Because contact reasoning relies heavily on the reconstructed scene geometry, any initial errors made by the scene reconstruction branch can propagate and degrade both contact estimation and human alignment.

Summary updated

Abstract

We introduce UniCon3R, a unified feed-forward framework for online human-scene 4D reconstruction from monocular video. Current feed-forward human-scene reconstruction methods suffer from artifacts, where bodies float above the ground or penetrate parts of the scene. A key reason is the lack of effective interaction modelling between the human and the environment. Our goal is to exploit contact between the human and the scene during inference to actively improve the human mesh reconstruction. To that end, we explicitly model interaction by inferring 4D contact from the human pose and scene geometry and use the contact as a corrective cue for generating the pose. This enables UniCon3R to jointly recover scene geometry and spatially aligned 4D humans within the scene. Experiments on standard human-centric video benchmarks show that UniCon3R outperforms state-of-the-art baselines on physical plausibility and global human motion estimation while preserving fast, feed-forward inference speeds. The results validate our central claim: contact serves as a powerful internal prior, thus establishing a new paradigm for physically grounded joint human-scene reconstruction. Project page is available at https://surtantheta.github.io/UniCon3R .
BibTeX
@misc{sur2026unicon,
  author = {Tanuj Sur and Shashank Tripathi and Nikos Athanasiou and Ha Linh Nguyen and Kai Xu and Michael J. Black and Angela Yao},
  title = {{UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video}},
  year = {2026},
  eprint = {2604.19923},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.19923},
}

Loading related content...

Figure