Abstract: Human motion synthesis in 3D scenes relies heavily on scene comprehension, while current methods focus mainly on scene structure but ignore the semantic understanding. In this paper, we propose a human motion synthesis framework that take an unified Scene Semantic Occupancy (SSO) for scene representation, termed SSOMotion. We design a bi-directional tri-plane decomposition to derive a compact version of the SSO, and scene semantics are mapped to an unified feature space via CLIP encoding and shared linear dimensionality reduction. Such strategy can derive the fine-grained scene semantic structures while significantly reduce redundant computations. We further take these scene hints and movement direction derived from instructions for motion control via frame-wise scene query. Extensive experiments and ablation studies conducted on cluttered scenes using ShapeNet furniture, as well as scanned scenes from PROX and Replica datasets, demonstrate its cutting-edge performance while validating its effectiveness and generalization ability.
Introduction
Generating realistic human motion that interacts naturally with a 3D environment—such as sitting on a chair or navigating around a table—is a dual challenge of geometry and semantics. Existing methods often fall into one of two traps: they either treat the world as a purely geometric collection of points, ignoring the functional meaning of objects (e.g., a sofa is for lying down, not just an obstacle), or they suffer from extreme computational complexity when processing full 3D semantic occupancy grids.
Researchers have developed SSOMotion, a new framework that allows AI to synthesize human movement by teaching it not just where objects are, but what they are. By compressing complex 3D scene data into a lightweight representation and unifying object semantics, the model achieves state-of-the-art performance with a staggering 99.9% reduction in computational overhead compared to raw 3D processing.
Methodology
The SSOMotion framework introduces three key innovations to bridge the gap between scene perception and motion generation:
- Bi-directional Tri-plane Decomposition: Instead of forcing the model to process a massive 3D grid, the system projects the 3D Scene Semantic Occupancy (SSO) onto three 2D planes ($xy, yz, zx$) from both directions. This captures the "shell" of the environment—depth, color, and semantics—in a compact format, reducing costs from 20,783 GFLOPs to just 0.49 GFLOPs.
- Unified Semantic Space via CLIP: To solve the problem of inconsistent labeling across different datasets, the authors use a pre-trained CLIP text encoder. By mapping category names (like "stool" or "bench") into a shared feature space, the model gains a universal understanding of objects that transcends specific dataset definitions.
- Goal-Directed Human Scene Correlation: The framework uses a diffusion-based motion branch and a control branch. The control branch employs cross-attention to correlate the human’s current state with the scene’s semantic hints, ensuring the generated motion is physically plausible and contextually appropriate.
Experiments and Results
The model was trained and evaluated using high-quality motion datasets (AMASS, Babel, HumanML3D) and scene-aware datasets (HUMANISE). Evaluation spanned cluttered scenes using ShapeNet furniture and real-world scanned environments from the PROX and Replica datasets to test cross-dataset generalization.
- Precision and Plausibility: In locomotion tasks, SSOMotion achieved an average destination distance of just 0.02m, outperforming previous models like DIMOS (0.04m). It also showed significantly lower penetration (clipping through furniture) during complex interactions like sitting or lying down.
- Efficiency: The combination of tri-plane decomposition and dimensionality reduction made the scene comprehension component nearly 42,000 times more efficient than baseline 3D processing.
- User Preference: In user studies, SSOMotion was consistently rated higher than competitors across naturalness, diversity, and goal achievement.
Conclusion and Future Work
SSOMotion demonstrates that semantic understanding is a primary driver of intent in motion synthesis. By using language as a bridge via CLIP embeddings, the model generalizes effectively to new environments. While the current framework faces minor hurdles like voxelization artifacts (causing slight "hovering") and is limited to single-agent scenarios, it sets a new benchmark for efficient, scene-aware motion generation. Future research will focus on scaling to crowd generation and adapting to dynamic environments where furniture or other agents move in real-time.
Summary updated