TL;DR
SGIP (Scene-Grounded Interaction Prior) improves human motion prediction by explicitly forecasting where in a 3D environment a person is headed—whether to a specific object or through open space. By using a clever "privileged distillation" technique, the researchers move expensive LLM-based reasoning to the training phase, allowing a lightweight model to make highly accurate, scene-aware predictions at inference using only raw motion and 3D geometry.
What Problem Does This Paper Address?
Predicting how a human will move through a room is a fundamental challenge for autonomous robots and embodied agents. While we’ve gotten better at modeling basic physics, future-motion ambiguity remains a major hurdle.
Imagine a person walking toward a dining area. Are they going to sit in the chair on the left? The one on the right? Or are they just passing through to the kitchen? In a cluttered scene, the same initial movement could lead to dozens of valid destinations. Existing methods often try to solve this by using "gaze" data (tracking where the person is looking), but specialized eye-tracking hardware is rarely available in real-world deployments. Without gaze or explicit intent, models often "drift" between targets or produce physically implausible "averaged" motions.
Methods
The authors introduce SGIP, a framework that turns scene-interaction reasoning into an explicit, supervisable task. The core components include:
- Scene Grounding Tokens: The 3D scene is decomposed into "tokens" representing individual object instances (via 3D segmentation) and a global "free space" token for navigation.
- Privileged Intent Distillation: This is the "secret sauce."
- The Teacher: During training, a Vision-Language Model (VLM) looks at the video to extract a textual intent (e.g., "take some rest"). A Large Language Model (LLM) then scores how well each scene token fits that intent. This semantic score is fused with a geometric score (based on the person's current trajectory) to create a "Teacher Prior."
- The Student: A fast, permutation-equivariant MLP learns to replicate this distribution using only the observed motion and 3D scene. The expensive LLM and textual intent are discarded after training.
- Prior-Conditioned Prediction: The model selects the most likely token and uses its spatial mask and semantic embedding to steer a two-stage motion predictor. It first plans a trajectory toward the target and then generates the full-body SMPL-X pose sequence.
Datasets
The model was evaluated on several key benchmarks for human-scene interaction:
- GIMO: A real-world indoor dataset featuring full-body Poses, 3D scenes, and gaze annotations (used only for baseline comparisons).
- PROX: A dataset of humans interacting in indoor environments, notably lacking gaze data, which highlights SGIP's ability to work without auxiliary sensors.
- HIK & HOI-M3: Used to test the model's performance in more complex, multi-human dynamic scenes.
Key Insights and Ideas
- Common Sense via Distillation: LLMs possess "common sense" (e.g., "you sit on sofas, not lamps"), but they are too slow for real-time motion forecasting. SGIP successfully "bakes" this semantic knowledge into a lightweight student model.
- Explicit vs. Implicit Grounding: By forcing the model to pick a target token, the authors prevent the "mode-averaging" problem where a model predicts a path that goes between two chairs because it can't decide which one to choose.
- Geometry Disambiguates Semantics: An LLM knows a person wants to "sit," but if there are five chairs, it can't help. The geometric compatibility score (alignment and distance) acts as the tie-breaker to identify the specific instance the person is approaching.
- Mode Covering: By using forward KL divergence during distillation, the student is forced to maintain the full distribution of possible targets, which is crucial for generating diverse "multi-future" predictions.
Key Results
SGIP sets a new state-of-the-art for observation-only motion prediction:
- Accuracy: On the GIMO dataset, SGIP reduced trajectory error by 93mm compared to the strongest observation-only baseline.
- Generalization: On the PROX dataset (which has no gaze data), it outperformed baselines by 83mm in destination accuracy.
- Multi-Future Plausibility: When asked to provide three possible futures, SGIP generated distinct, interpretable paths (e.g., walking to different chairs) that were significantly more accurate than those generated by diffusion-based models.
- Efficiency: The student model runs at ~7.5 FPS on a single RTX 3090, making it viable for real-time robotics applications.
Limitations and Future Work
- Segmentation Dependency: The system's performance is tied to the quality of the initial 3D instance segmentation. If the segmenter misses an object, the model cannot interact with it.
- Static Scene Assumption: While tested on multi-human data, the model currently treats other humans as static "tokens." A natural next step is modeling time-varying human and object states.
- One-time Pre-processing: The system requires a one-time segmentation pass for every new scene, which takes about a minute, potentially hindering immediate deployment in entirely unseen environments.
Summary updated