H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers

Wenhao Li, Mengyuan Liu, Hong Liu, Pichao Wang, Shijian Lu, Nicu Sebe

H2OT (Hierarchical Hourglass Tokenizer) is a plug-and-play framework designed to accelerate 3D human pose estimation from video. By "pruning" redundant video frames in intermediate Transformer layers and "recovering" them at the output, H2OT creates an "hourglass" architecture. This approach reduces computational costs (FLOPs) by up to 60% while maintaining or even improving accuracy across major benchmarks.

What Problem Does This Paper Address?

Estimating 3D human poses from 2D video sequences is critical for applications like action recognition and human-robot interaction. Video Pose Transformers (VPTs) have become the standard due to their ability to model long-range temporal dependencies. However, Transformers suffer from quadratic computational complexity relative to the sequence length. Processing long sequences (e.g., 243 frames) to achieve high accuracy results in a heavy "computational tax," making these models unsuitable for real-time deployment on edge devices.

The authors observed that video data is inherently redundant; consecutive frames often contain nearly identical information. Standard VPTs process every frame through every layer, wasting significant energy on redundant computations.

Comparison of VPT, HoT, and H2OT architectures. H2OT introduces a hierarchical pruning strategy to create a more efficient hourglass structure.

Methods

The H2OT framework addresses this inefficiency by progressively discarding redundant tokens (frames) and reconstructing them only when needed for the final output.

The H2OT framework pipeline: 2D poses are embedded and processed through initial Transformer blocks before entering the Token Pruning Module. The reduced set of tokens passes through core Transformer blocks, after which the Token Recovering Module restores the full sequence for 3D pose regression.

The framework consists of three primary components:

  1. Token Pruning Module (TPM): This module identifies and removes redundant frames. The paper evaluates several strategies, including Clustering (TPC), Attention-based (TPA), Motion-based (TPMo), and simple Uniform Sampling (TPS).
  2. Hierarchical Design: Unlike its predecessor (HoT), which pruned tokens in a single step, H2OT adopts a hierarchical approach. Tokens are discarded gradually across multiple layers, creating a "pyramidal" feature hierarchy. This allows the model to maintain a more nuanced representation of the temporal sequence while still reducing the total token count.
  3. Token Recovering Module (TRM): To ensure the model provides a 3D pose for every input frame—essential for smooth video output—H2OT employs a recovery module at the end of the pipeline. This can be achieved via Cross-Attention (TRA) or simple Linear Interpolation (TRI).

Key Results and Insights

H2OT was evaluated on the Human3.6M and MPI-INF-3DHP datasets, demonstrating significant efficiency gains across various baseline models:

  • MixSTE Integration: Reduced FLOPs by 57.4% and increased inference speed (FPS) by 87.8%. Notably, the error rate (MPJPE) improved by 0.5mm, suggesting that pruning can act as a regularizer by removing temporal noise.
  • MotionBERT Integration: Achieved a 63.4% reduction in FLOPs and a 47.6% boost in FPS with negligible impact on accuracy.
  • Evolution from HoT: Compared to the previous conference version (HoT), H2OT reduces GPU memory usage by approximately 26% and training time by 25%, thanks to the refined hierarchical pruning strategy.

The authors found that Uniform Sampling (TPS) combined with Linear Interpolation (TRI) often provided the best trade-off between speed and accuracy. This combination is parameter-free and highly robust, making it ideal for real-world deployment.

Limitations and Future Work

While H2OT offers substantial improvements, it faces challenges in specific scenarios:

  • Extreme Occlusions: Long-duration occlusions might lead the pruning module to discard the few frames where a body part is visible.
  • Rare Poses: Out-of-distribution poses may not be well-captured by the representative tokens.
  • 2D Detector Dependency: As a "lifting" approach, the final 3D accuracy is bounded by the quality of the initial 2D joint detections.

Future research may focus on end-to-end optimization that integrates 2D detection and 3D lifting into a single efficient pipeline, potentially improving robustness to occlusions.

Summary updated

BibTeX
@article{li2025h,
  author = {Wenhao Li and Mengyuan Liu and Hong Liu and Pichao Wang and Shijian Lu and Nicu Sebe},
  title = {{H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3608284},
}
Figure