Dual-Branch Graph Transformer Network for 3D Human Mesh Reconstruction from Video

Wenhao Li, Tao Tang, Hong Liu, Yingxuan You, Ti Wang, Wenhao Li 0002

This paper, titled "Dual-Branch Graph Transformer Network for 3D Human Mesh Reconstruction from Video," introduces a novel architecture designed to solve the persistent trade-off between accuracy and temporal smoothness in video-based human modeling.

TL;DR

The authors propose DGTR, a dual-branch network that simultaneously models long-term global motion and fine-grained local details using a combination of Transformers and Modulated Graph Convolutional Networks (GCNs). The result is a state-of-the-art model that is more accurate and smoother than previous methods while requiring significantly fewer parameters and computational cycles (FLOPs).

What Problem Does This Paper Address?

3D Human Mesh Reconstruction (HMR) involves estimating the full 3D shape and pose of a person from a 2D video. While image-based methods have become quite accurate, applying them to video is difficult.

Existing video-based methods usually fall into two traps:

  1. The RNN Trap: Using Recurrent Neural Networks (like GRUs) often leads to "over-smoothing," where the model loses the "crispness" of movement, resulting in inaccurate meshes.
  2. The Attention Trap: While Transformers are excellent at capturing global temporal dependencies (the "big picture" of a movement), they often miss "local crucial information"—the tiny, subtle movements or joint interactions that happen across just a few frames.

The challenge is building a network that understands the entire 16-frame sequence (global) without losing the nuanced details of what's happening between frame 7 and frame 8 (local).

Methods

The core innovation of DGTR is its Dual-Branch architecture, which processes a sequence of static features (extracted via a ResNet50 backbone) in parallel:

  • Global Motion Attention (GMA) Branch: This uses a lightweight, 2-layer Transformer encoder. Its job is to look at the entire input sequence (typically 16 frames) to ensure the overall motion is smooth and physically plausible over time.
  • Local Details Refine (LDR) Branch: This branch focuses on the "nitty-gritty." It consists of two parts:
  • Local Information Aggregation: A 1D convolution layer that slides over a small window (3 frames) to aggregate local details and reduce redundancy.
  • Crucial Feature Extraction: Instead of standard self-attention, this uses a Modulated Graph Convolutional Network (MGCN). By treating video frames as nodes in a graph, the MGCN uses learnable "weight modulation" to capture strong correlations between adjacent frames, effectively "refining" the local details that a global transformer might ignore.

Finally, the features from both branches are summed and fed into a standard SMPL parameter regressor to produce the final 3D mesh.

Datasets

The model was trained and evaluated on the most rigorous benchmarks in the field:

  • 3DPW (3D Pose in the Wild): The primary benchmark for "in-the-wild" performance.
  • Human3.6M: A massive indoor dataset with precise ground truth.
  • MPI-INF-3DHP: A dataset featuring diverse indoor and outdoor scenes.
  • InstaVariety & AMASS: Used for additional training data and motion priors.

Key Insights and Ideas

  • Parallelism is Key: Previous methods often stacked temporal filters serially. By running global and local branches in parallel, DGTR prevents the global "smoothing" effect from washing out the local "details."
  • GCNs as "Local" Transformers: The authors found that replacing standard multi-head attention with a Modulated GCN in the local branch significantly reduced the number of parameters while improving the model's ability to capture tiny movements.
  • Efficiency Matters: Unlike many SOTA models that grow increasingly bloated, DGTR is remarkably lean. It achieves better results with roughly 10.89M parameters, compared to TCMR’s 50.43M or VIBE’s 15.01M.

Key Results

DGTR sets a new bar for efficiency and accuracy:

  • Accuracy: On the 3DPW dataset, it reduced the Mean Per Joint Position Error (MPJPE) to 82.0mm, a significant improvement over the previous leader, MPS-Net (84.3mm).
  • Smoothness: It achieved the best acceleration error (a measure of jitter) on the MPI-INF-3DHP dataset and remained highly competitive on others.
  • Computational Efficiency: DGTR uses the fewest FLOPs (277.56M) among all compared video-based methods, making it a strong candidate for real-time applications like robotics or AR/VR.

Limitations and Future Work

The authors identify one primary bottleneck: severe occlusion. When a human body is significantly blocked by objects or cut off by the camera frame (truncation), the model’s performance degrades.

Future work will likely involve making the dual-branch architecture more robust to these "missing data" scenarios, perhaps by incorporating more sophisticated spatial-temporal priors to "guess" the position of occluded limbs.

Summary updated

Abstract

Human Mesh Reconstruction (HMR) from monocular video plays an important role in human-robot interaction and collaboration. However, existing video-based human mesh reconstruction methods face a trade-off between accurate reconstruction and smooth motion. These methods design networks based on either RNNs or attention mechanisms to extract local temporal correlations or global temporal dependencies, but the lack of complementary long-term information and local details limits their performance. To address this problem, we propose a \textbf{D}ual-branch \textbf{G}raph \textbf{T}ransformer network for 3D human mesh \textbf{R}econstruction from video, named DGTR. DGTR employs a dual-branch network including a Global Motion Attention (GMA) branch and a Local Details Refine (LDR) branch to parallelly extract long-term dependencies and local crucial information, helping model global human motion and local human details (e.g., local motion, tiny movement). Specifically, GMA utilizes a global transformer to model long-term human motion. LDR combines modulated graph convolutional networks and the transformer framework to aggregate local information in adjacent frames and extract crucial information of human details. Experiments demonstrate that our DGTR outperforms state-of-the-art video-based methods in reconstruction accuracy and maintains competitive motion smoothness. Moreover, DGTR utilizes fewer parameters and FLOPs, which validate the effectiveness and efficiency of the proposed DGTR. Code is publicly available at \href{https://github.com/TangTao-PKU/DGTR}{\textcolor{myBlue}{https://github.com/TangTao-PKU/DGTR}}.
BibTeX
@inproceedings{li2024dual,
  author = {Wenhao Li and Tao Tang and Hong Liu and Yingxuan You and Ti Wang and Wenhao Li},
  title = {{Dual-Branch Graph Transformer Network for 3D Human Mesh Reconstruction from Video}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801569},
}

Loading related content...

Figure