Lifting Motion to 3D Scene Flow

143 concepts107 papers1 proposal

Traditional computer vision models often represent movement as optical flow: a two-dimensional field of pixel displacements. While effective for simple tasks, 2D flow suffers from fundamental geometric ambiguities. For instance, a pixel moving toward the camera and a pixel scaling up due to a zoom lens can appear identical in a 2D projection. To resolve these ambiguities, a growing body of research focuses on lifting motion into three-dimensional space, a representation known as scene flow.

The Ambiguity of the 2D Plane

The primary limitation of 2D motion modeling is its inability to distinguish between changes in an object's position and changes in its scale or perspective. Rahul Malik et al. argue that incorporating depth-axis flow is critical for distinguishing actual movement toward a sensor from simple scaling effects, particularly in immersive environments. Without this third dimension, models often struggle with "radial" movements—actions occurring along the camera's Z-axis—which the authors of a 2019 study on localized trajectories note are frequently lost in traditional projection-based tracking.

Furthermore, 2D representations are susceptible to perspective distortion. Fan Yang et al. demonstrate that tracking objects in 3D coordinates allows for more accurate motion prediction because it preserves real-scale motion. In 3D, a person walking at a constant speed maintains a linear trajectory, whereas in 2D, their apparent speed changes based on their distance from the lens.

Lifting Pixels to 3D Trajectories

To bridge the gap between 2D pixels and 3D space, researchers "lift" motion using depth information. This depth can come from stereo sequences, as explored by Frédéric Huguet et al., or from monocular depth estimation. Yuxi Xiao et al. show that lifting 2D video data into a 3D coordinate system using monocular depth allows for robust tracking through complex out-of-plane rotations, converting intricate pixel movements into regular 3D trajectories.

A key mechanism for this lifting is optical expansion. Gengshan Yang et al. propose "upgrading" optical flow to scene flow by estimating local scale changes (how much an object grows or shrinks) alongside pixel displacement. This allows the model to calculate a 3D motion vector that identifies the true direction of motion in world space. Similarly, Hanhan Li et al. demonstrate that feeding predicted depth maps into motion estimation networks enables models to reason about displacements in 3D coordinates, significantly improving accuracy in high-traffic, dynamic environments.

3D-Aware Generation and Control

The shift toward 3D-aware motion is also transforming generative AI. Standard video generation often suffers from "physical phasing" (objects passing through each other) or jittery motion because the model lacks a spatial blueprint. Taehoon Kim et al. address this by first generating a 3D human motion sequence and then projecting it into 2D skeletons. This intermediate 3D step provides a structural guide that prevents disappearing limbs and incorrect scaling.

In the realm of user control, Hanlin Wang et al. introduce LeviTor, which integrates depth estimation into user-defined motion paths. By transforming 2D "drag-and-drop" trajectories into 3D spatial coordinates, the system can handle complex movements like orbiting and realistic occlusions. For robotics, Yang Bai et al. show that using depth-encoded 3D trajectories prevents generative models from creating physically impossible scenarios, such as a robot arm passing through a solid object.

Holistic 3D Motion Understanding

Beyond simple tracking, lifting motion to 3D enables a more holistic understanding of scene geometry. Zhenheng Yang et al. developed a "Holistic 3D Motion Parser" (HMP) that uses per-pixel 3D scene flow to distinguish between an object's physical distance and its independent velocity. This resolves "projective confusion," where a moving object at one depth might appear identical to a static object at another.

This 3D grounding is also essential for reconstructing dynamic scenes. Tae-Hyun Oh et al. use Neural Scene Flow Fields (NSFF) to represent dynamic scenes as continuous functions of 3D space and time. By modeling motion as 3D vectors within a volumetric representation, they can synthesize novel camera angles and sharp motion boundaries that 2D methods cannot maintain. Similarly, Cameron Omid Smith et al. lift 2D optical flow to 3D scene flow to train generalizable radiance fields (NeRFs) without requiring pre-calculated camera poses, demonstrating that 3D-aware motion is a powerful signal for joint geometry and pose reconstruction.

Key Foundational Work

ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis (1998) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_depth_aware_motion_modeling_8452c9e2,
  title = {depth-aware motion modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-aware-motion-modeling/rdr_8452c9e2fe8e49b8b977b4fdc5ee85db},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_depth_aware_motion_modeling_8452c9e2_v1,
  title = {depth-aware motion modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-aware-motion-modeling/rdr_8452c9e2fe8e49b8b977b4fdc5ee85db/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake