Scene-Aware 3D Human Motion Prediction

This direction is split into other directions: environment-agnostic motion modeling. Its last version stays readable and citable here.

Version 1 of 177 papers91 conceptsfrozen 2026-10-04

Traditional human motion prediction (HMP) models often treat the human body as a collection of joints moving in a vacuum. While these models can learn the basic kinematics of a walk or a wave, they suffer from "environmental blindness." Without knowledge of the surrounding world, an AI might predict a person walking through a solid wall, floating above a chair, or "drifting" into a static, average pose that ignores the purpose of the movement.

The research direction emerging from these papers shifts the focus from pure kinematics to scene-aware forecasting. The core intuition is that human movement is not just a sequence of coordinates, but a continuous negotiation with the physical environment.

The Problem of Environmental Blindness

Most early deep learning models for HMP, such as those discussed by Minjing Dong and Chang Xu, rely on historical skeletal data to extrapolate future frames. However, as Zhenguang Liu and Haipeng Chen point out, this "tunnel vision" fails to account for external factors.

When a model is unaware of a table or a doorway, it generates "ghost motion"—trajectories that are mathematically smooth but physically impossible. Furthermore, without a goal (like a chair to sit on), long-term predictions often collapse into a "mean pose," where the character slowly stops moving because the model cannot "see" the reason to continue.

Modeling Constraints and Affordances

To solve this, researchers have introduced architectures that treat the environment as a set of constraints (what you cannot do) and affordances (what you can do).

Enric Corona and Albert Pumarola pioneered the use of semantic graphs to bridge this gap. By representing the human and nearby objects as nodes in a graph, the model learns how the proximity of an object—like a cup or a handle—dictates the trajectory of the hand. This transforms motion prediction from a regression task into an interaction task.

Taking this a step further, Chaoyue Xing and Wei Mao proposed predicting the mutual distance between body parts and scene surfaces. Instead of just guessing the next joint position, the model predicts how far each limb is from the floor or furniture. This provides a "whole-body constraint" that prevents the character from penetrating solid geometry.

Intent and Gaze as Contextual Cues

If the environment provides the "where," human intent provides the "why." Recent work has begun integrating human perception into the forecasting loop. Ting Yu introduced the GAP3DS framework, which uses human gaze to inform motion. By understanding what a person is looking at, the model can infer their intent—distinguishing, for example, between someone walking past a chair and someone intending to sit in it.

Similarly, María Escobar explores egocentric (first-person) perspectives, where visual cues from the wearer's environment are fused with head trajectories to predict body movements in unscripted, "in the wild" scenarios.

Bridging the Data Gap

A major hurdle in this field is that traditional datasets, like Human3.6M, were recorded in empty studios. To train scene-aware models, researchers need data where the environment is as well-documented as the person.

Jiaman Li addressed this with CIRCLE, a system using Virtual Reality to capture human motion in complex, cluttered environments. This allows models to learn how humans naturally adjust their posture to avoid obstacles or navigate cramped spaces, providing the rich contextual data necessary for the next generation of physically grounded AI.

From Vacuum to Reality

The shift toward scene-awareness represents a maturation of the field. By moving away from isolated joint kinematics and toward frameworks that respect 3D geometry—such as the Encoding-Alignment-Interaction (EAI) framework by Pengxiang Ding or the Multi-Task Graph Convolutional Networks of Qiongjie Cui—researchers are creating models that don't just predict movement, but understand the physical world that movement inhabits.

Papers in this version

Showing 60 of 77 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scene_aware_3d_human_motion_prediction_d07c3033,
  title = {Scene-Aware 3D Human Motion Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scene-aware-3d-human-motion-prediction/rdr_d07c30330dfc4542ad8e62e55a50226f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scene_aware_3d_human_motion_prediction_d07c3033_v1,
  title = {Scene-Aware 3D Human Motion Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scene-aware-3d-human-motion-prediction/rdr_d07c30330dfc4542ad8e62e55a50226f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}