Reconstructing Human Motion in Non Planar Environments

This direction is split into other directions: planar ground assumption (flat-ground assumption). Its last version stays readable and citable here.

Version 1 of 165 papers68 conceptsfrozen 2026-10-04

For decades, computer vision and robotics have relied on a convenient mathematical lie: the world is flat. When estimating 3D human motion or navigating a robot, researchers often assume the agent moves on an infinite, perfectly level 2D plane. This "flat ground assumption" simplifies the math of depth estimation and camera calibration, but it collapses the moment a person steps onto a curb, climbs a staircase, or walks up a hill.

Recent research is moving beyond this constraint, developing systems that can simultaneously reconstruct human motion and the complex, non-planar terrain beneath it.

The Failure of the Flat Plane

In traditional 3D reconstruction, the ground plane acts as a "global anchor." If you know where the floor is, you can resolve the ambiguity of a single camera: a person who appears smaller is simply further away on that plane. However, as noted by Saburo Tsuji and Jiang Yu Zheng as early as 1987, these fixed geometric relationships fail on uneven terrain.

When a model assumes a flat floor but the real world has stairs or slopes, the resulting 3D motion suffers from two distinct artifacts:

  1. Footskate and Penetration: The character’s feet appear to slide across the surface or "sink" through the floor geometry.
  2. Vertical Drift: Without a way to account for elevation changes, systems like those described by Jiaxi Jiang and Christian Holz struggle to maintain accurate height-based motion cues during multi-story transitions.

Joint Reconstruction of Motion and Terrain

The most direct solution to the flat-plane problem is to stop treating the environment as a constant and start treating it as a variable. Instead of projecting humans onto a pre-defined floor, new frameworks reconstruct the human and the scene geometry simultaneously.

Yifeng Jiang and Yuting Ye demonstrated this with the Transformer Inertial Poser (TIP), which uses sparse IMU data to reconstruct motion while generating the surrounding terrain in real-time. By predicting where the ground must be to support the observed limb accelerations, the system can accurately capture a person sitting on a chair or climbing stairs without any visual input of the furniture itself. Similarly, Crowd4D by Hongbo Kang and colleagues explicitly models the interaction between crowd trajectories and non-planar surfaces like curbs and transit hubs to eliminate "floating" artifacts.

Humans as Metric Rulers

One of the most powerful tools for overcoming non-planar terrain is the human body itself. Because we know the statistical distribution of human heights and the proportions of our limbs, a person in a video acts as a "metric ruler" for the scene.

Lea Müller, Jitendra Malik, and their team at UC Berkeley developed Humans and Structure from Motion (HSfM) to leverage this intuition. By using human scale to calibrate camera parameters, they can recover the absolute metric scale of a scene—even when the subject is jumping or climbing—where traditional Structure-from-Motion (SfM) would fail. This synergy is also explored in SynCHMR, which uses human dimensions to calibrate SLAM systems in cluttered, non-planar indoor environments.

Grounding Motion in Physics

If geometry is ambiguous, physics is not. Researchers are increasingly using "intuitive physics" to ground motion in complex environments. If a foot is stationary and experiencing a high upward force, there must be a supporting surface there, regardless of whether the camera can see a flat floor.

Shashank Tripathi and Lea Müller introduced IPMAN, which integrates differentiable physics terms like Center of Mass and Center of Pressure into pose estimation. This ensures that reconstructed bodies are physically stable and supported by the actual scene geometry, rather than just visually aligned with pixels. Davis Rempe and Bryan C. Russell furthered this by using centroidal trajectory optimization to estimate ground reaction forces, allowing for plausible motion even when the contact points are occluded or the terrain is irregular.

Egocentric and Sparse Sensing

The push for "in-the-wild" motion capture has moved these solutions into wearable form factors. Systems like EgoHDM and the work of Hanbyul Joo combine head-mounted cameras with sparse sensors (like smartwatches) to track motion across hills and stairs. By fusing SLAM-based head tracking with floor-level update algorithms, these systems can maintain a global coordinate frame that survives the transition from a flat living room to a steep outdoor trail.

Papers in this version

Showing 60 of 65 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reconstructing_human_motion_in_non_plana_716a6e43,
  title = {Reconstructing Human Motion in Non Planar Environments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reconstructing-human-motion-in-non-planar-environments/rdr_716a6e43daec476bbff247bc57cc1b0e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_reconstructing_human_motion_in_non_plana_716a6e43_v1,
  title = {Reconstructing Human Motion in Non Planar Environments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reconstructing-human-motion-in-non-planar-environments/rdr_716a6e43daec476bbff247bc57cc1b0e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}