Abstract
Monocular 3D human pose estimation (3DHPE) often suffers from depth ambiguity, where multiple 3D configurations can project to the same 2D coordinates. This paper introduces the Two-step Mixed-Training (TMT) strategy, which leverages 3D joint velocity as an explicit input feature during training. By alternating between real 3D velocity and zero-vector proxies, the model learns robust spatio-temporal features that persist during inference even when 3D data is unavailable. Experimental results on Human3.6M and MPI-INF-3DHP datasets demonstrate a significant reduction in Mean Per Joint Position Error (MPJPE), achieving up to a 13.8% improvement over baseline architectures.
Introduction
The primary challenge in monocular 3DHPE is "lifting" 2D coordinates into 3D space. While temporal information is crucial for resolving depth ambiguities, existing methods primarily use joint velocity as a loss constraint rather than a direct input. This research proposes using 3D velocity as "privileged information" during training to enhance the model's feature representation capabilities early in the processing pipeline.
Methodology: Two-step Mixed-Training (TMT)
The TMT strategy is designed for Transformer-based lifting models. It expands the standard 2D input $\vec{p_2} = (u, w)$ into a 5D vector by appending three dimensions for 3D velocity $\vec{v_3} = (\Delta x, \Delta y, \Delta z)$.
As illustrated in the framework above, the training process alternates between two distinct steps:
- Step 1 (Inference Proxy): The model receives 2D positions concatenated with a 3D zero vector $\vec{z_3} = (0, 0, 0)$. This prepares the model for real-world deployment where 3D velocity is unknown.
- Step 2 (Privileged Learning): The model receives 2D positions concatenated with the actual 3D velocity $\vec{v_3}$ derived from ground truth.
This alternating optimization forces the model to extract deeper spatio-temporal relationships when velocity is present, while the zero-padding in Step 1 acts as a regularizer, ensuring the model remains functional using only 2D inputs during deployment.
Experimental Results and Insights
The TMT strategy was evaluated on the Human3.6M and MPI-INF-3DHP benchmarks. On Human3.6M, applying TMT to a MixSTE backbone reduced the MPJPE from 21.6mm to 18.6mm—a relative improvement of 13.8%.
A key finding of this study is the direct positive correlation between joint velocity and estimation error. As shown in the joint-wise analysis, fast-moving joints like the wrists and feet exhibit significantly higher MPJPE. By explicitly providing velocity data during training, TMT helps the model better resolve the trajectories of these high-velocity joints. Furthermore, the strategy improves robustness against occlusions, as the learned temporal features allow the model to better estimate the positions of hidden joints based on their motion history.
Discussion and Conclusion
TMT demonstrates that moving temporal information from the loss function to the input layer significantly enhances feature learning in early network stages. While the strategy is highly effective, its gains are partially dependent on the quality of the initial 2D detections. Future work may explore more complex integration mechanisms beyond simple concatenation or the use of predicted 2D velocity as an inference-time proxy. In conclusion, TMT offers a simple yet powerful training paradigm that substantially advances the state-of-the-art in 3D human pose estimation by effectively bridging the gap between privileged training data and real-world inference constraints.
Summary updated