Abstract
Human Mesh Reconstruction (HMR) from monocular video plays an important role in human-robot interaction and collaboration. However, existing video-based human mesh reconstruction methods face a trade-off between accurate reconstruction and smooth motion. These methods design networks based on either RNNs or attention mechanisms to extract local temporal correlations or global temporal dependencies, but the lack of complementary long-term information and local details limits their performance. To address this problem, we propose a Dual-branch Graph Transformer network for 3D human mesh Reconstruction from video, named DGTR. DGTR employs a dual-branch network including a Global Motion Attention (GMA) branch and a Local Details Refine (LDR) branch to parallelly extract long-term dependencies and local crucial information, helping model global human motion and local human details (e.g., local motion, tiny movement). Specifically, GMA utilizes a global transformer to model long-term human motion. LDR combines modulated graph convolutional networks and the transformer framework to aggregate local information in adjacent frames and extract crucial information of human details. Experiments demonstrate that our DGTR outperforms state-of-the-art video-based methods in reconstruction accuracy and maintains competitive motion smoothness. Moreover, DGTR utilizes fewer parameters and FLOPs, which validate the effectiveness and efficiency of the proposed DGTR.
Introduction
3D Human Mesh Reconstruction (HMR) involves estimating the full 3D shape and pose of a person from a 2D video source. While image-based methods have reached high levels of accuracy for static frames, extending these techniques to video introduces significant challenges regarding temporal consistency.
Existing video-based methods typically encounter two primary limitations. First, models relying on Recurrent Neural Networks (RNNs) often suffer from "over-smoothing," where the resulting motion loses the crispness of real-world movement. Second, while Transformers are excellent at capturing global temporal dependencies, they often overlook "local crucial information"—the subtle joint interactions and tiny movements that occur across just a few frames. The challenge lies in building a network that understands the entire sequence (global context) without sacrificing the nuanced details of frame-to-frame transitions (local context).
The DGTR Architecture
To resolve the trade-off between global smoothness and local accuracy, we introduce DGTR, a dual-branch architecture that processes static features extracted via a ResNet50 backbone in parallel.
Global Motion Attention (GMA) Branch
The GMA branch utilizes a lightweight, two-layer Transformer encoder. Its primary function is to analyze the entire input sequence (typically 16 frames) to ensure the overall motion is physically plausible and temporally smooth. By modeling long-term dependencies, the GMA provides a stable foundation for the human mesh over time.
Local Details Refine (LDR) Branch
The LDR branch focuses on fine-grained movements through two specific components:
- Local Information Aggregation: A 1D convolution layer slides over a small three-frame window to aggregate local details and reduce data redundancy.
- Crucial Feature Extraction: Instead of standard self-attention, this branch employs a Modulated Graph Convolutional Network (MGCN). By treating video frames as nodes in a graph, the MGCN uses learnable weight modulation to capture strong correlations between adjacent frames, effectively refining the local details that a global transformer might otherwise ignore.
The features from both the GMA and LDR branches are fused via summation and passed to a standard SMPL parameter regressor to generate the final 3D mesh.
Experimental Evaluation
DGTR was evaluated against the most rigorous benchmarks in the field, including 3DPW (3D Pose in the Wild), Human3.6M, and MPI-INF-3DHP. For training, the model utilized additional motion priors from the InstaVariety and AMASS datasets.
Key Results and Efficiency
DGTR establishes a new state-of-the-art for both accuracy and computational efficiency:
- Accuracy: On the 3DPW dataset, DGTR achieved a Mean Per Joint Position Error (MPJPE) of 82.0mm, outperforming previous leading models like MPS-Net (84.3mm).
- Smoothness: The model achieved the best acceleration error (a metric for jitter) on the MPI-INF-3DHP dataset, demonstrating that accuracy does not have to come at the cost of motion stability.
- Computational Overhead: DGTR is remarkably lean, utilizing only 10.89M parameters and 277.56M FLOPs. This is a significant reduction compared to other SOTA models like TCMR (50.43M parameters), making DGTR highly suitable for real-time applications in robotics and AR/VR.
Conclusion and Future Work
The dual-branch approach of DGTR successfully addresses the limitations of previous temporal modeling techniques by processing global and local information in parallel. While the model shows superior performance, it still faces challenges in scenarios involving severe occlusion or camera truncation. Future research will focus on enhancing the architecture's robustness to missing data, potentially by incorporating more sophisticated spatial-temporal priors to estimate the positions of occluded limbs.
Summary updated