Unified Pose Sequence Modeling

Lin Geng Foo, Tianjiao Li, Hossein Rahmani, Qiuhong Ke, Jun LiuCVPR 2023 · OpenReview

Abstract

We propose a Unified Pose Sequence Modeling approach to unify heterogeneous human behavior understanding tasks based on pose data, e.g., action recognition, 3D pose estimation and 3D early action prediction. A major obstacle is that different pose-based tasks require different output data formats. Specifically, the action recognition and prediction tasks require class predictions as outputs, while 3D pose estimation requires a human pose output, which limits existing methods to leverage task-specific network architectures for each task. Hence, in this paper, we propose a novel Unified Pose Sequence (UPS) model to unify heterogeneous output formats for the aforementioned tasks by considering text-based action labels and coordinate-based human poses as language sequences. Then, by optimizing a single auto-regressive transformer, we can obtain a unified output sequence that can handle all the aforementioned tasks. Moreover, to avoid the interference brought by the heterogeneity between different tasks, a dynamic routing mechanism is also proposed to empower our UPS with the ability to learn which subsets of parameters should be shared among different tasks. To evaluate the efficacy of the proposed UPS, extensive experiments are conducted on four different tasks with four popular behavior understanding benchmarks.

Introduction

Human behavior understanding has traditionally been a fragmented field in computer vision. Different tasks—such as action recognition, 3D pose estimation, and early action prediction—require distinct output formats. Action recognition typically outputs a single categorical label, whereas 3D pose estimation requires a continuous sequence of numerical joint coordinates. This heterogeneity has forced researchers to develop specialized "heads" and architectures for each task, hindering the development of a generalist model for human motion. Furthermore, joint training on multiple tasks often suffers from "destructive interference," where performance on one task degrades while learning another.

Methodology

The Unified Pose Sequence (UPS) model addresses these challenges by treating human pose data as a language. The framework consists of three primary components: a Joint Tokenizer, a UPS Encoder, and a UPS Decoder.

The overall architecture of the Unified Pose Sequence (UPS) model, showing the flow from input skeletons to task-specific output tokens.

Unified Vocabulary and Tokenization

To bridge the gap between categorical labels and continuous coordinates, UPS converts all outputs into a unified token sequence. Action labels (e.g., "jumping") are mapped to text embeddings using a pre-trained RoBERTa model. Continuous joint coordinates are quantized into discrete bins and associated with text-based descriptions, such as "the first horizontal coordinate." This allows the model to treat all tasks as sequence generation problems.

The process of converting tasks, coordinates, and actions into unified target tokens using text prompts and a pre-trained language model.

Architecture and Dynamic Routing

The Joint Tokenizer uses a lightweight GCN-based module to convert raw input coordinates into latent tokens. The UPS Encoder employs Spatial Encoding Modules (SEM) and Temporal Encoding Modules (TEM) to capture body topology and temporal dynamics. To mitigate task interference, a dynamic routing mechanism utilizes task embeddings and Gumbel-Softmax routing to select specific parameter blocks for each task, enabling selective knowledge sharing. Finally, the UPS Decoder, a standard auto-regressive transformer, predicts the output sequence until an "End of Task" (EOT) token is reached.

Results

The UPS model was evaluated on four major benchmarks: NTU RGB+D 60 & 120, Kinetics 400, and Human3.6M.

  • 3D Action Recognition: UPS achieved 92.6% (xsub) on NTU 60, outperforming many specialized GCN-based methods.
  • 2D Action Recognition: On Kinetics 400, it reached 40.5% Top-1 accuracy, a significant improvement over previous GCN-based models (~37%).
  • 3D Pose Estimation: The model achieved an error of 40.8mm (MPJPE) on Human3.6M, remaining highly competitive with specialized lifting models.
  • Early Action Prediction: With only 20% of video observation, UPS achieved 53.25% accuracy, significantly higher than the ~42% achieved by specialized methods.

Conclusion

By treating human motion as a structured language, the UPS framework demonstrates that a single model can effectively unify diverse behavior understanding tasks. While auto-regressive decoding presents some challenges for real-time inference, the ability to selectively share knowledge across tasks through dynamic routing allows UPS to match or exceed the performance of specialized architectures. Future work may extend this unified approach to human-object interactions and full-body mesh recovery.

Summary updated

Abstract

We propose a Unified Pose Sequence Modeling approach to unify heterogeneous human behavior understanding tasks based on pose data, e.g., action recognition, 3D pose estimation and 3D early action prediction. A major obstacle is that different pose-based tasks require different output data formats. Specifically, the action recognition and prediction tasks require class predictions as outputs, while 3D pose estimation requires a human pose output, which limits existing methods to leverage task-specific network architectures for each task. Hence, in this paper, we propose a novel Unified Pose Sequence (UPS) model to unify heterogeneous output formats for the aforementioned tasks by considering text-based action labels and coordinate-based human poses as language sequences. Then, by optimizing a single auto-regressive transformer, we can obtain a unified output sequence that can handle all the aforementioned tasks. Moreover, to avoid the interference brought by the heterogeneity between different tasks, a dynamic routing mechanism is also proposed to empower our UPS with the ability to learn which subsets of parameters should be shared among different tasks. To evaluate the efficacy of the proposed UPS, extensive experiments are conducted on four different tasks with four popular behavior understanding benchmarks.
BibTeX
@inproceedings{foo2023unified,
  author = {Lin Geng Foo and Tianjiao Li and Hossein Rahmani and Qiuhong Ke and Jun Liu},
  title = {{Unified Pose Sequence Modeling}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=P2ifdHpWfp},
}
Figure