Abstract
We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.
Introduction
Estimating a person's full-body 3D pose from a head-mounted camera is an inherently ill-posed problem. Because the camera is mounted on the head, the wearer’s own body is rarely visible, leaving researchers to solve a puzzle with most of the pieces missing. Traditional computer vision approaches, or kinematics-based methods, attempt to map video features directly to joint angles. While these can produce visually reasonable results in isolation, they often suffer from physical inconsistencies such as "foot skating," floating motions, or interpenetration with scene objects because the models lack an understanding of gravity and physical constraints.
Conversely, purely physics-based methods (dynamics) attempt to drive a humanoid within a simulator. While these produce realistic movement, they are difficult to control and prone to failure when visual input is noisy. Furthermore, most existing work fails to account for how the wearer interacts with environmental objects, such as sitting on a chair or pushing a box.
Methodology: Dynamics-Regulated Training
To bridge the gap between pixels and physics, we introduce a hybrid framework that synergizes kinematics and dynamics through Dynamics-Regulated Training (DRT). The system consists of three core components:
- Universal Humanoid Controller (UHC): A low-level, task-agnostic controller pre-trained on the AMASS motion capture dataset. Its role is to execute movements that match a target pose within a physics simulator without losing balance.
- Kinematic Policy: A high-level planner that analyzes egocentric video, camera trajectory, and the 6DoF state of scene objects to propose the next pose.
- The Feedback Loop: During training, the Kinematic Policy proposes a pose, which the UHC then attempts to mimic inside the MuJoCo physics simulator. If the proposed pose is physically impossible—for instance, if it violates gravity or collision constraints—the humanoid fails to match the target. This misalignment generates a physics-based error signal used to refine the Kinematic Policy via Reinforcement Learning (RL).
By using the simulator as a non-differentiable "critic," the model learns to propose poses that are both visually consistent with the video and physically achievable in the real world.
Experimental Evaluation
We evaluated our method using three primary data sources: the AMASS database for UHC pre-training, a new MoCap dataset consisting of 266 sequences (148k frames) of specific human-object interactions (sitting, pushing, stepping, etc.), and a real-world dataset captured using a VR-mounted smartphone.
Quantitative and Qualitative Results
Our approach demonstrates a significant leap over previous state-of-the-art methods such as EgoPose and PoseReg:
- Accuracy: On the MoCap dataset, our model reduced root position error by 79.4% and joint position error by 62.1% compared to purely kinematic baselines.
- Physical Plausibility: The model achieved a human-object interaction success rate of 96.9%. In contrast, previous physics-based models succeeded only 48.4% of the time, frequently failing during complex tasks like stepping onto a box.
- Generalization: The model successfully estimated poses from "in-the-wild" footage, proving that the learned physics-aware policy is robust to diverse real-world environments.
Conclusion
By integrating physics simulation directly into the training loop of a kinematic policy, we have enabled the first physically-plausible estimation of human-object interactions from a single wearable camera. While the system currently requires prior knowledge of object categories and sizes, it represents a major step toward robust, physics-aware egocentric perception. Future work will focus on scaling the system to handle arbitrary objects and more complex, unconstrained scene interactions.
Summary updated