Understanding how objects move in three dimensions is a cornerstone of autonomous driving, robotics, and digital content creation. However, obtaining ground-truth labels for 3D motion—such as scene flow or volumetric fluid dynamics—is notoriously difficult and expensive. While 3D sensors like LiDAR and Radar provide spatial context, they are often sparse or physically limited in what they can measure.
A growing body of research addresses this by using 2D optical flow as a "universal teacher." By bridging the gap between 2D pixels and 3D space through geometry and differentiable rendering, these methods allow models to learn complex 3D dynamics from unlabeled video sequences.
The Differentiable Bridge
The core intuition behind this research direction is that 2D motion is a projection of 3D motion. If a model predicts how a 3D scene changes over time, that prediction can be projected back into the 2D image plane. The discrepancy between this projection and the observed 2D optical flow provides a supervisory signal to refine the 3D model.
Mao et al. demonstrate this by using a differentiable renderer to supervise 3D occupancy flow. By comparing 2D perception cues like segmentation and optical flow against rendered outputs, the model learns to predict 3D dynamics without any manual 3D annotations. Similarly, Franz et al. and the authors of Neural Global Transport apply this to fluid dynamics, using a differentiable ray-marching renderer to reconstruct 3D smoke motion from a single 2D video. In the realm of deformable objects, Zhan et al. show that 3D cloth dynamics can be recovered from multi-view videos by backpropagating image reconstruction errors into geometric updates.
Resolving Physical Sensor Limits
This cross-modal approach is particularly powerful for sensors that are physically incapable of capturing full motion vectors. For instance, Doppler radar only measures radial velocity—how fast an object moves toward or away from the sensor—leaving tangential ("sideways") motion unknown.
Long et al. and Liu et al. resolve this by fusing 1D radar returns with 2D camera optical flow. By treating these as complementary geometric constraints, they use a closed-form solution to recover the full 3D velocity vector. A similar logic applies to event cameras, which suffer from the "aperture problem" where a single moving line cannot reveal full motion. Gao et al. and Gehrig et al. address this by fusing partial velocity observations from multiple features into a unified 3D linear velocity estimate.
Regularizing Sparse 3D Data
LiDAR sensors provide precise depth but are often spatially sparse, making it hard to distinguish between static backgrounds and moving objects. Dense 2D optical flow can act as a high-resolution regularizer for these sparse 3D points.
Luo et al. introduced a framework where dense 2D flow provides "free" supervision for the motion of 3D "pillars" in a bird’s-eye view. This allows the model to leverage pixel-level motion to fill in the gaps where LiDAR data is missing. Lahlali et al. further show that 2D motion cues are often more reliable than 3D scene flow for discovering objects in the first place, using 2D flow to guide the segmentation of objects in sparse LiDAR point clouds. Even in decentralized settings, Fang et al. demonstrate that cross-modal guidance helps roadside units distinguish moving traffic from static infrastructure despite sensor noise.
Distilling Knowledge from Foundation Models
As 2D vision models become more powerful, they can serve as "teachers" for 3D systems. Instead of relying on raw video, researchers are now distilling motion knowledge directly from pre-trained foundation models.
Zhao et al. use this approach in SplatFlow, where a 3D motion field is refined by minimizing the discrepancy between its projections and the outputs of a 2D optical flow foundation model. This results in sharper reconstructions of dynamic scenes. In the domain of character animation, Ma et al. leverage 2D generative models to produce target expressions, using the resulting 2D displacement fields to supervise the auto-rigging of complex 3D facial meshes.
By treating 2D motion as a projection of a higher-dimensional reality, these methods turn standard video into a rich, automated laboratory for learning the physics of the 3D world.
Key Foundational Work
Self-Supervised Pillar Motion Learning for Autonomous Driving (2021) — contributed 2 concept elements to this direction.
3D Motion Decomposition for RGBD Future Dynamic Scene Synthesis (2019) — contributed 2 concept elements to this direction.
Full-Velocity Radar Returns by Radar-Camera Fusion (2021) — contributed 2 concept elements to this direction.