Abstract
Monocular 3D human pose estimation faces a fundamental challenge: the inherent ambiguity of depth. Traditional "lifting" methods often entangle reliable 2D pose features with uncertain depth information, leading to a phenomenon termed "depth erosion," where Z-axis instability degrades X-Y accuracy. We present PoseMoE, a Mixture-of-Experts (MoE) architecture designed to decouple 2D pose refinement from depth discovery. By utilizing specialized experts and a cross-attention-based decoder for strategic aggregation, PoseMoE achieves state-of-the-art performance on Human3.6M and 3DPW benchmarks while demonstrating superior robustness to input noise.
Introduction
Monocular 3D human pose estimation—the task of predicting 3D joint positions from a single 2D image—is a cornerstone of computer vision with applications in robotics, animation, and healthcare. However, the transition from 2D to 3D is inherently "ill-posed" because multiple 3D configurations can project onto the same 2D plane.
Most modern "lifting" frameworks first detect 2D joints and then map them to 3D space. A critical, often overlooked issue in these models is "depth erosion." When 2D coordinates and unknown depth are projected into a shared feature space, the high uncertainty of the depth dimension propagates into the reliable 2D features. Empirical evidence suggests that in traditional models, high depth error is almost always correlated with a drop in 2D accuracy, sometimes resulting in 3D-to-2D projections that are less accurate than the original 2D inputs. PoseMoE is designed to solve this by treating 2D refinement and depth discovery as distinct tasks.
Methodology: The PoseMoE Framework
To address depth erosion, PoseMoE adopts a "decouple then aggregate" strategy. The architecture is divided into two primary components: the PoseMoE Encoder (PME) and the PoseMoE Decoder (PMD).
PoseMoE Encoder (PME)
Instead of a uniform processing block, the PME employs specialized experts to handle the different "initial states" of 2D and 3D data:
- 2D Pose Expert: This module focuses on refining the detected 2D coordinates by leveraging structural human constraints. It treats the input as a refinement task rather than a discovery task.
- Depth Expert: This module addresses the missing Z-dimension. Since depth is initially unknown, this expert utilizes learnable tokens initialized from a Gaussian distribution. Grounded in the Central Limit Theorem, this provides a statistically stable prior that facilitates better convergence compared to zero or Laplace initialization.
PoseMoE Decoder (PMD)
After the experts have purified their respective features in isolation, the PMD performs "Cross-Expert Knowledge Aggregation." Using bidirectional spatial and temporal cross-attention, the decoder allows the 2D pose and depth features to inform one another. This ensures that the final 3D estimation benefits from 2D structural consistency without allowing depth uncertainty to corrupt the 2D features during the early encoding stages. Mutual Information (MI) validation confirms that the encoder maintains low MI (successful decoupling), while the decoder achieves high MI (successful strategic fusion).
Experimental Evaluation
PoseMoE was evaluated on three major benchmarks: Human3.6M (indoor), MPI-INF-3DHP (complex/outdoor), and 3DPW (real-world/outdoor).
Performance and Efficiency
PoseMoE sets new state-of-the-art (SOTA) records across major benchmarks:
- Human3.6M: Achieved a Mean Per Joint Position Error (MPJPE) of 38.7mm, outperforming powerful models like KTPFormer and MixSTE.
- 3DPW: Achieved an MPJPE of 76.8mm, a significant improvement over the previous best of 81.0mm.
- Temporal Efficiency: The model demonstrates high efficiency; a version using 81 input frames outperformed several competing models that required 243 or even 351 frames.
Robustness to Noise
When synthetic noise was added to the 2D input, PoseMoE showed a much shallower performance drop compared to other lifting-based methods. This proves that the 2D Pose Expert successfully shields the model from the "depth erosion" that typically occurs when noisy 2D inputs are processed in entangled architectures.
Conclusion and Future Work
PoseMoE demonstrates that strategic decoupling is essential for solving the ill-posed problem of 3D pose lifting. By treating 2D refinement and depth discovery as distinct tasks and aggregating them only after feature purification, the model effectively mitigates the depth erosion effect.
While the model handles self-occlusion well, extreme occlusion or severe motion blur in the input 2D pose can still lead to failures. Future research will focus on extending this MoE approach to multi-person scenarios, addressing inter-person depth ambiguity, and integrating explicit uncertainty estimation directly into the framework to further refine the knowledge aggregation process.
Summary updated