Uplift and Upsample: Efficient 3D Human Pose Estimation With Uplifting Transformers

Moritz Einfalt, Katja Ludwig, Rainer LienhartWACV 2023

TL;DR

This paper introduces a novel Transformer-based approach for efficient 3D human pose estimation from monocular video. By decoupling the input 2D pose sampling rate from the output 3D pose rate, their method significantly reduces computational complexity, enabling real-time performance. They also demonstrate the benefits of pre-training on motion capture data for improved accuracy.

What Problem Does This Paper Address?

Estimating 3D human pose from a single 2D camera (monocular 3D HPE) is a challenging but crucial task with applications ranging from animation to action recognition. The dominant paradigm for this is "2D-to-3D pose uplifting," where a 2D pose estimator first detects human joints in each frame, and then a separate model "uplifts" these 2D poses into 3D. While the uplifting models themselves can be efficient, the overall computational bottleneck lies in the per-frame 2D pose estimation, especially for high-resolution videos. This makes real-time 3D HPE on resource-constrained devices like smartphones difficult. The paper addresses this by proposing a method that can achieve dense 3D pose predictions from temporally sparse 2D pose inputs, thereby drastically reducing the computational cost of the initial 2D estimation.

Methods

The core of the proposed method is a Transformer architecture designed for joint uplifting and temporal upsampling. Instead of processing every 2D pose from the video, the system takes 2D poses from sparse key-frames. These sparse 2D poses are first transformed into pose tokens. To bridge the temporal gap and generate dense 3D poses, the authors introduce a "learnable upsampling token." This token acts as a placeholder for missing 2D poses and is integrated into the Transformer's self-attention mechanism. Through masked token modeling, the Transformer learns to predict 3D poses for all frames within its temporal receptive field, effectively upsampling the sparse input.

The architecture consists of three main components:

  1. Joint-Wise Spatial Transformer: This module processes each input 2D pose individually, using self-attention across joints to create a robust pose representation.
  2. Pose-Wise Temporal Transformer: This is the core upsampling module. It takes the sequence of pose tokens (including the upsampling tokens) and uses self-attention to generate dense temporal 3D pose predictions.
  3. Strided Transformer: This module is used for refining the central frame's prediction and reducing the sequence length efficiently, enabling a symmetric N-to-1 prediction scheme during evaluation.

The model is trained with a combined loss function that includes a sequence loss (encouraging temporally stable predictions) and a center frame loss (minimizing error for the central predicted pose). Additionally, the authors explore pre-training the Transformer on large-scale motion capture datasets (AMASS) to improve performance, especially when dealing with sparse inputs.

Datasets

The paper evaluates its method on two standard 3D HPE datasets:

  • Human3.6M: A widely used dataset for indoor 3D HPE, consisting of 11 actors performing 15 actions, recorded by four stationary RGB cameras at 50 Hz.
  • MPI-INF-3DHP: A more challenging dataset for single-person 3D HPE, featuring greater diversity in motion, viewpoints, and environments, with recordings from a single camera.

The authors also leverage the AMASS meta-dataset for pre-training. AMASS is a collection of motion capture data that is re-targeted to generate detailed 3D mesh models.

Key Insights and Ideas

  • Decoupling Input and Output Rates is Key: The most significant insight is that by separating the rate of 2D pose input from the desired 3D pose output rate, computational complexity can be drastically reduced without sacrificing accuracy. This is achieved by intelligently filling in the temporal gaps.
  • Learnable Upsampling Tokens: The introduction of learnable upsampling tokens within the Transformer is a clever way to handle missing temporal information. These tokens are not just placeholders but actively contribute to learning the temporal dynamics.
  • Pre-training on Unpaired Motion Capture Data is Beneficial: The paper highlights that pre-training on large, unpaired motion capture datasets can significantly boost the performance of Transformer-based uplifting models, even compensating for the challenges posed by sparse input sequences.
  • Deeper Architectures Benefit from Sparsity: The proposed deeper Transformer architecture, when combined with sparse inputs, shows a smaller performance drop compared to shallower models, indicating that deeper networks can better leverage the available information.

Key Results

  • Significant Speedup: The method achieves a speed-up factor of 12x in total computation (including 2D pose estimation) compared to state-of-the-art methods when using an input stride of 20 (i.e., processing only 1/20th of the frames for 2D estimation). This enables real-time throughput on consumer GPUs.
  • Competitive Accuracy: Despite the reduced input rate, the proposed method achieves competitive spatial precision. On Human3.6M, it reaches an MPJPE of 45.0 mm (with MoCap pre-training and sparse input), comparable to state-of-the-art methods. On MPI-INF-3DHP, it achieves the third-best PCK, AUC, and MPJPE scores.
  • Improved Temporal Consistency: The full-sequence supervision and the explicit upsampling mechanism lead to smoother and more temporally consistent 3D pose predictions compared to simple bilinear interpolation.
  • Flexibility: The model supports variable input frame rates during inference without requiring retraining, making it adaptable to different applications and hardware constraints.

Limitations and Future Work

  • Accuracy Trade-off with Extreme Sparsity: While the method handles sparsity well, extremely sparse input sequences (e.g., 2.5 Hz input for 50 Hz output) can lead to a slight increase in MPJPE, though this is largely mitigated by pre-training.
  • Reliance on 2D Pose Estimator: The overall performance is still dependent on the quality of the initial 2D pose estimation. Errors in the 2D poses will propagate to the 3D predictions.
  • Future Work: The authors suggest further exploring the benefits of pre-training on even larger datasets and investigating more advanced data augmentation techniques. They also mention that the flexibility of the model opens up possibilities for real-time applications on mobile devices.

Summary updated

BibTeX
@inproceedings{einfalt2023uplift,
  author = {Moritz Einfalt and Katja Ludwig and Rainer Lienhart},
  title = {{Uplift and Upsample: Efficient 3D Human Pose Estimation With Uplifting Transformers}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure