The Chicken-and-Egg Problem of Dynamic Video
Imagine watching a casual smartphone video of someone walking toward the camera. As they approach, their silhouette expands in the frame. Did the person step forward, or did the camera zoom in? If they shift to the left, did they move sideways, or did the camera pan right?
In computer vision, this is a classic "chicken-and-egg" ambiguity. Disentangling the movement of a dynamic subject from the movement of the camera—and mapping both accurately within a static 3D environment—is incredibly difficult from a single video feed.
Traditionally, researchers split this challenge into isolated pipelines:
- Structure from Motion (SfM) or SLAM: Algorithms tracked background features to estimate camera trajectories and static scene geometry, treating moving humans as outliers or noise to be ignored.
- Human Mesh Recovery (HMR): Algorithms tracked the human body, but only within an isolated, camera-centric coordinate frame.
This sequential approach introduces severe physical inconsistencies. Because the human model has no awareness of the scene geometry, and the scene reconstruction has no awareness of the human, the final combined output suffers from glaring artifacts. Characters appear to float in mid-air, slide across the floor, or clip through solid walls. Furthermore, monocular reconstructions lack an absolute scale, meaning the reconstructed human might be sized like a giant or a doll relative to the room.
The Core Insight: Unified Optimization
To resolve these ambiguities, a powerful paradigm shift has emerged: joint optimization. Instead of treating human tracking and scene reconstruction as independent steps, these variables are coupled into a single, unified objective function.
The core intuition is that the human and the environment are not independent; they mutually constrain one another. As demonstrated by Muhammed Kocabas et al. in PACE: Human and Camera Motion Estimation from in-the-wild Videos, optimizing human kinetics and scene geometry simultaneously allows the system to satisfy both visual evidence (2D keypoints and background features) and physical motion priors at the same time. The certainties of one variable resolve the ambiguities of another.
The Human Body as a Metric Ruler
One of the most elegant properties of joint optimization is its ability to solve the classic monocular scale ambiguity. In standard geometric reconstruction, a scene can only be recovered up to an arbitrary scale factor unless an object of known size is present.
Joint optimization frameworks realize that the human body itself is that object. Human proportions and bone lengths follow predictable statistical distributions.
[2D Video Frame] ───► [Joint Optimization Loop] ◄─── [Human Kinematic Prior]
│
├───► Establishes Absolute Metric Scale
├───► Disentangles Camera vs. Subject Motion
└───► Enforces Physical Contact (No Floating)
Lea Müller et al. exploit this in Reconstructing People, Places, and Cameras by using the known statistical size of the human body as a "metric ruler." By minimizing 2D projection errors for human joints alongside alignment errors for dense scene pointmaps, the optimization unlocks the absolute metric scale of the entire 3D environment.
Similarly, Rishabh Dabral et al. utilize bone-length ratios in Multi-Person 3D Human Pose Estimation from Monocular Images to resolve global positioning, successfully placing multiple independently detected subjects into a single, unified 3D camera coordinate system.
Disentangling Motion via Kinematic Priors
To separate how much a person moves from how much the camera moves, joint frameworks leverage sophisticated temporal and kinematic priors. If a camera pans rapidly, the background features and the human will shift across the frame together. If the human moves independently, their joints will change configuration in a way that obeys human anatomy.
For instance, Jiefeng Li et al. introduce a diffusion-based motion prior in COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation to optimize human pose, camera trajectory, and global scale factors together. Because the prior understands natural human physics, it penalizes physically impossible accelerations. If the visual data shows a sudden shift that violates human kinetics, the optimization attributes that movement to the camera trajectory instead of the actor.
This principle applies across diverse sensor modalities and rendering techniques:
- Inertial Sensors: Timo von Marcard et al. combine moving cameras with wearable IMUs in Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera, balancing 2D visual projection errors against 3D inertial measurements to jointly solve for both the camera's path and the human's trajectory.
- Neural Radiance Fields (NeRFs): Anyi Rao et al. build a differentiable pipeline in Cinematic Behavior Transfer via NeRF-based Differentiable Filming that optimizes character motion and camera trajectories simultaneously, resolving conflicts between dynamic subject movement and static background tracking to enable high-fidelity cinematic transfers.
Physical Grounding through Contact Constraints
When humans and scenes are reconstructed independently, the human often appears detached from reality. Joint optimization eliminates this by treating human-scene interaction as a physical bridge.
In Joint Optimization for 4D Human-Scene Reconstruction in the Wild, Zhizheng Liu et al. enforce explicit human-scene contact constraints. Because the camera, human, and scene parameters are optimized in a single stage, the floor geometry adjusts to meet the feet, and the feet adjust to stay planted, eliminating artifacts like foot sliding.
This holistic grounding is crucial for downstream tasks. Lixin Xue et al. demonstrate in HSR: Holistic 3D Human-Scene Reconstruction from Monocular Videos that simultaneously optimizing for both dynamic human models and static scene geometry within a unified coordinate system eliminates spatial artifacts like floating or clipping.
This environmental alignment is what allows robots to learn from casual videos; Arthur Allshire et al. utilize this in Visual Imitation Enables Contextual Humanoid Control to prevent physical inconsistencies like interpenetration, ensuring that reconstructed movements are contextually aligned so humanoids can learn to sit on chairs or climb stairs.
For real-time applications, Zetong Zhang et al. leverage 3D Gaussian Splatting in ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos to jointly optimize camera trajectories and human poses within a unified representation, matching the tracking accuracy of specialized SLAM systems.
Scaling to Uncalibrated Multi-View Environments
The power of joint optimization extends naturally to multi-view scenarios where camera calibration is completely unknown or highly dynamic. While traditional visual SLAM frameworks, such as the one described by Yujian Pan et al. in Robust Direct Multi-Camera SLAM in Challenging Scenarios, rely strictly on tightly coupled multi-camera geometric and photometric constraints to map static environments, incorporating human pose tracking directly into the bundle adjustment loop provides an entirely new set of dynamic anchors.
Instead of relying on rigid calibration patterns like checkerboards, moving people themselves can serve as the calibration target. Ko Nishino et al. demonstrate in Spatiotemporal Multi-Camera Calibration Using Freely Moving People that aligning 3D human poses across uncalibrated views unifies camera calibration, synchronization, and person association into a single registration problem.
By coupling these variables, systems become highly resilient to noise and occlusions:
- Uncalibrated Streams: Buzhen Huang et al. in Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras and Roman Bachmann et al. in Motion Capture from Pan-Tilt Cameras with Unknown Orientation minimize joint loss functions combining 2D projections and temporal motion priors, preventing the optimization from falling into degenerate local minima.
- Internet Videos: Even when videos are completely unsynchronized and collected from different sources, Junting Dong et al. show in Motion Capture from Internet Videos that a bundle adjustment framework can simultaneously refine camera extrinsics and 3D human body parameters across a "pseudo-multi-view" setup.
- Fine-Grained Personalization: This multi-frame lifting strategy is also vital for fine-grained tracking. David Joseph Tan et al. optimize hand shape parameters across multiple depth frames simultaneously in Fits Like a Glove: Rapid and Reliable Hand Shape Personalization to prevent overfitting to a single pose.
- Total Capture: On a larger scale, Hanbyul Joo formulates markerless motion capture in Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies as a unified optimization problem balancing 2D keypoints and 3D point clouds across faces, hands, and bodies.
- Challenging Environments: In specialized environments like operating rooms, Abdolrahim Kadkhodamohammadi et al. demonstrate in A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms that minimizing an energy function accounting for detection confidence and depth-based reprojection ensures that the tracked human remains physically consistent with the observed environmental surfaces.
By breaking down the walls between camera tracking, scene reconstruction, and human pose estimation, joint optimization transforms a series of error-prone, isolated tasks into a cooperative mathematical framework where every piece of visual data helps anchor the rest.
Key Foundational Work
A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms (2017) — contributed 1 concept elements to this direction.
Fits Like a Glove: Rapid and Reliable Hand Shape Personalization (2016) — contributed 1 concept elements to this direction.