SceMoS: Scene-Aware 3D Human Motion Synthesis by Planning with Geometry-Grounded Tokens

A. Ghosh, Vladislav Golyanik, Taku Komura, Philipp Slusallek, Christian Theobalt, Rishabh Dabral

Abstract

Synthesizing text-driven 3D human motion within realistic scenes requires learning both semantic intent ("walk to the couch") and physical feasibility (e.g., avoiding collisions). Current methods use generative frameworks that simultaneously learn high-level planning and low-level contact reasoning, and rely on computationally expensive 3D scene data such as point clouds or voxel occupancy grids. We propose SceMoS, a scene-aware motion synthesis framework that shows that structured 2D scene representations can serve as a powerful alternative to full 3D supervision in physically grounded motion synthesis. SceMoS disentangles global planning from local execution using lightweight 2D cues and relying on (1) a text-conditioned autoregressive global motion planner that operates on a bird's-eye-view (BEV) image rendered from an elevated corner of the scene, encoded with DINOv2 features, as the scene representation, and (2) a geometry-grounded motion tokenizer trained via a conditional VQ-VAE, that uses 2D local scene heightmap, thus embedding surface physics directly into a discrete vocabulary. This 2D factorization reaches an efficiency-fidelity trade-off: BEV semantics capture spatial layout and affordance for global reasoning, while local heightmaps enforce fine-grained physical adherence without full 3D volumetric reasoning. SceMoS achieves state-of-the-art motion realism and contact accuracy on the TRUMANS benchmark, reducing the number of trainable parameters for scene encoding by over 50%, showing that 2D scene cues can effectively ground 3D human-scene interaction.

Introduction

Synthesizing human motion that is both semantically correct (e.g., "walk to the table and sit") and physically plausible (avoiding collisions and making proper contact with furniture) is a significant challenge in computer vision. Existing approaches typically suffer from two main issues: scene-agnosticism, where characters float or clip through furniture, and computational bloat, caused by the use of dense 3D representations like voxel grids or point clouds that require massive backbones.

SceMoS Overview: The framework takes BEV images and text prompts as input to generate physically grounded 3D motion.

SceMoS addresses these challenges by asking whether full 3D volumetric reasoning is truly necessary. Instead, it leverages structured 2D projections—Bird’s-Eye-View (BEV) images and local heightmaps—to achieve high-fidelity Human-Scene Interaction (HSI) with significantly reduced computational overhead.

Methodology

SceMoS introduces a disentangled two-stage approach that separates high-level planning from low-level physical execution.

1. Global Motion Planner

The global planner acts as the "brain," deciding where the character should go. It utilizes an autoregressive transformer conditioned on text prompts (encoded via T5) and global scene context. Rather than a full 3D scan, it uses a single BEV image rendered from an elevated corner, processed by a frozen DINOv2 encoder. This provides rich semantic features regarding the room's layout and affordances.

2. Geometry-Grounded Motion Tokenizer

The tokenizer acts as the "body," deciding how to move. A conditional VQ-VAE is trained to learn a discrete vocabulary of motion tokens. Crucially, the decoder is conditioned on a local 2D heightmap centered on the character's current position. By grounding motion tokens in local geometry, the model learns to align character joints (like hips) with the height of surfaces, embedding physics directly into the motion vocabulary.

Detailed architecture of the Global Motion Planner (a) and the Geometry-Grounded Motion Tokenizer (b).

3. Inference Loop

During inference, the system iteratively re-renders the BEV and re-calculates the local heightmap as the character moves. This allows for long-range, multi-step motions while maintaining physical consistency throughout the sequence.

Key Results and Insights

SceMoS was evaluated on the TRUMANS benchmark (15 hours of motion in 100 scenes) and the HUMANISE dataset for generalization. Key findings include:

  • State-of-the-Art Realism: SceMoS achieved a record-low FID of 0.31 on TRUMANS, indicating superior similarity to real human data.
  • Physical Accuracy: The model reached a Contact Score of 0.98, effectively eliminating artifacts like floor penetration or floating.
  • Efficiency: By replacing 3D voxels with 2D cues, the framework requires over 50% fewer trainable parameters for scene encoding compared to voxel-based baselines. The entire pipeline can generate 80 frames of motion in approximately 8 seconds.
  • Foundation Model Synergy: Leveraging frozen vision foundation models like DINOv2 allows the planner to understand object affordances without requiring massive amounts of labeled 3D data.

Conclusion

SceMoS demonstrates that 2D projections are "good enough" for complex 3D human-scene interaction. By disentangling planning from execution and grounding motion in local geometry, the framework achieves a new state-of-the-art in realism and efficiency. Future work will focus on adapting the model for dynamic scenes with moving obstacles and fine-grained object manipulation.

Summary updated

Abstract

Synthesizing text-driven 3D human motion within realistic scenes requires learning both semantic intent ("walk to the couch") and physical feasibility (e.g., avoiding collisions). Current methods use generative frameworks that simultaneously learn high-level planning and low-level contact reasoning, and rely on computationally expensive 3D scene data such as point clouds or voxel occupancy grids. We propose SceMoS, a scene-aware motion synthesis framework that shows that structured 2D scene representations can serve as a powerful alternative to full 3D supervision in physically grounded motion synthesis. SceMoS disentangles global planning from local execution using lightweight 2D cues and relying on (1) a text-conditioned autoregressive global motion planner that operates on a bird's-eye-view (BEV) image rendered from an elevated corner of the scene, encoded with DINOv2 features, as the scene representation, and (2) a geometry-grounded motion tokenizer trained via a conditional VQ-VAE, that uses 2D local scene heightmap, thus embedding surface physics directly into a discrete vocabulary. This 2D factorization reaches an efficiency-fidelity trade-off: BEV semantics capture spatial layout and affordance for global reasoning, while local heightmaps enforce fine-grained physical adherence without full 3D volumetric reasoning. SceMoS achieves state-of-the-art motion realism and contact accuracy on the TRUMANS benchmark, reducing the number of trainable parameters for scene encoding by over 50%, showing that 2D scene cues can effectively ground 3D human-scene interaction.
BibTeX
@inproceedings{ghosh2026scemos,
  author = {A. Ghosh and Vladislav Golyanik and Taku Komura and Philipp Slusallek and Christian Theobalt and Rishabh Dabral},
  title = {{SceMoS: Scene-Aware 3D Human Motion Synthesis by Planning with Geometry-Grounded Tokens}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=VT0g2PDLzG},
}
Figure