SGIP: Learning Scene-Grounded Interaction Priors for Scene-Aware Human Motion Prediction

Donghyeon Soon, Jiil Yoon, Hyeondong Kim, Daehee ParkarXiv 2026 · arXiv

TL;DR

SGIP (Scene-Grounded Interaction Prior) improves human motion prediction by explicitly forecasting where in a 3D environment a person is headed—whether to a specific object or through open space. By using a clever "privileged distillation" technique, the researchers move expensive LLM-based reasoning to the training phase, allowing a lightweight model to make highly accurate, scene-aware predictions at inference using only raw motion and 3D geometry.

What Problem Does This Paper Address?

Predicting how a human will move through a room is a fundamental challenge for autonomous robots and embodied agents. While we’ve gotten better at modeling basic physics, future-motion ambiguity remains a major hurdle.

Imagine a person walking toward a dining area. Are they going to sit in the chair on the left? The one on the right? Or are they just passing through to the kitchen? In a cluttered scene, the same initial movement could lead to dozens of valid destinations. Existing methods often try to solve this by using "gaze" data (tracking where the person is looking), but specialized eye-tracking hardware is rarely available in real-world deployments. Without gaze or explicit intent, models often "drift" between targets or produce physically implausible "averaged" motions.

Methods

The authors introduce SGIP, a framework that turns scene-interaction reasoning into an explicit, supervisable task. The core components include:

  1. Scene Grounding Tokens: The 3D scene is decomposed into "tokens" representing individual object instances (via 3D segmentation) and a global "free space" token for navigation.
  2. Privileged Intent Distillation: This is the "secret sauce."
    • The Teacher: During training, a Vision-Language Model (VLM) looks at the video to extract a textual intent (e.g., "take some rest"). A Large Language Model (LLM) then scores how well each scene token fits that intent. This semantic score is fused with a geometric score (based on the person's current trajectory) to create a "Teacher Prior."
    • The Student: A fast, permutation-equivariant MLP learns to replicate this distribution using only the observed motion and 3D scene. The expensive LLM and textual intent are discarded after training.
  3. Prior-Conditioned Prediction: The model selects the most likely token and uses its spatial mask and semantic embedding to steer a two-stage motion predictor. It first plans a trajectory toward the target and then generates the full-body SMPL-X pose sequence.

Datasets

The model was evaluated on several key benchmarks for human-scene interaction:

  • GIMO: A real-world indoor dataset featuring full-body Poses, 3D scenes, and gaze annotations (used only for baseline comparisons).
  • PROX: A dataset of humans interacting in indoor environments, notably lacking gaze data, which highlights SGIP's ability to work without auxiliary sensors.
  • HIK & HOI-M3: Used to test the model's performance in more complex, multi-human dynamic scenes.

Key Insights and Ideas

  • Common Sense via Distillation: LLMs possess "common sense" (e.g., "you sit on sofas, not lamps"), but they are too slow for real-time motion forecasting. SGIP successfully "bakes" this semantic knowledge into a lightweight student model.
  • Explicit vs. Implicit Grounding: By forcing the model to pick a target token, the authors prevent the "mode-averaging" problem where a model predicts a path that goes between two chairs because it can't decide which one to choose.
  • Geometry Disambiguates Semantics: An LLM knows a person wants to "sit," but if there are five chairs, it can't help. The geometric compatibility score (alignment and distance) acts as the tie-breaker to identify the specific instance the person is approaching.
  • Mode Covering: By using forward KL divergence during distillation, the student is forced to maintain the full distribution of possible targets, which is crucial for generating diverse "multi-future" predictions.

Key Results

SGIP sets a new state-of-the-art for observation-only motion prediction:

  • Accuracy: On the GIMO dataset, SGIP reduced trajectory error by 93mm compared to the strongest observation-only baseline.
  • Generalization: On the PROX dataset (which has no gaze data), it outperformed baselines by 83mm in destination accuracy.
  • Multi-Future Plausibility: When asked to provide three possible futures, SGIP generated distinct, interpretable paths (e.g., walking to different chairs) that were significantly more accurate than those generated by diffusion-based models.
  • Efficiency: The student model runs at ~7.5 FPS on a single RTX 3090, making it viable for real-time robotics applications.

Limitations and Future Work

  • Segmentation Dependency: The system's performance is tied to the quality of the initial 3D instance segmentation. If the segmenter misses an object, the model cannot interact with it.
  • Static Scene Assumption: While tested on multi-human data, the model currently treats other humans as static "tokens." A natural next step is modeling time-varying human and object states.
  • One-time Pre-processing: The system requires a one-time segmentation pass for every new scene, which takes about a minute, potentially hindering immediate deployment in entirely unseen environments.

Summary updated

Abstract

Human motion prediction (HMP) aims to forecast future movement from an observed motion sequence, and recent methods have incorporated 3D scene geometry and object semantics to improve long-term plausibility. However, the destination of future motion remains ambiguous in cluttered scenes: the same observed motion can lead to different interaction targets or navigation modes. Existing methods either handle this ambiguity implicitly or rely on auxiliary signals like gaze that are hard to obtain in practice. To address this, we propose SGIP, which explicitly predicts where in the scene future motion will be grounded across both objects and open navigation regions, and uses it to condition future trajectory and full-body pose prediction. This scene-grounded interaction prior is learned via intent distillation: at training time only, a teacher model scores candidate scene regions by combining an LLM prompted with a textual description of the person's intent with geometric motion-scene features. A student model learns to reproduce these scores from only the observed motion and 3D scene, making intent and LLM unnecessary at test time. Across multiple benchmark datasets, SGIP outperforms prior scene-aware baselines on both trajectory and pose accuracy in a realistic deployment setting where neither gaze nor intent is available at inference, demonstrating the effectiveness of the proposed framework. The project page is available at https://sgip2026.github.io.
BibTeX
@misc{soon2026sgip,
  author = {Donghyeon Soon and Jiil Yoon and Hyeondong Kim and Daehee Park},
  title = {{SGIP: Learning Scene-Grounded Interaction Priors for Scene-Aware Human Motion Prediction}},
  year = {2026},
  eprint = {2611.20076},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20076},
}
Figure