Human-Centered Prior-Guided and Task-Dependent Multi-Task Representation Learning for Action Recognition Pre-Training

Guanhong Wang, Keyu Lu, Yang Zhou, Zhanhao He, Gaoang Wang

Introduction

Self-supervised learning (SSL) has emerged as a powerful paradigm for action recognition, enabling models to learn robust video representations from vast amounts of unlabeled data. By using pretext tasks—such as predicting video playback speed or temporal order—models can capture essential features without the need for manual annotation. However, existing SSL methods often struggle with two primary limitations: the learning of "neutral" representations that lack semantic depth, and performance degradation caused by conflicting objectives in multi-task learning (MTL) frameworks.

This paper introduces a novel pre-training framework designed to address these challenges by incorporating human-centered prior knowledge and a task-dependent architecture. By focusing on the human actors who drive the majority of actions in video data, the proposed method achieves state-of-the-art performance on standard benchmarks.

The Limitations of Current SSL Methods

Most current self-supervised approaches rely on contrastive learning, which emphasizes instance-level similarity. While effective at capturing appearance and motion consistency, this approach often results in neutral representations. These features are often too generic, failing to capture the specific semantic nuances—such as human posture and body part movement—required for complex action recognition.

Furthermore, many frameworks attempt to learn multiple pretext tasks simultaneously using a single shared representation. This multi-task conflict occurs because different tasks (e.g., motion prediction versus appearance matching) often have inconsistent optimization goals. When forced into a single feature vector, these tasks can "fight" each other, leading to sub-optimal solutions that hinder the model's overall performance.

The Proposed Framework: Human-Centered and Task-Dependent

To overcome these hurdles, the authors propose a framework that integrates human-centric semantic information and a specialized architecture to manage multiple learning objectives.

1. Human-Centered Prior Knowledge Distillation

Recognizing that humans are the focal point of action recognition, the framework utilizes a pre-trained Human Parsing model as a "teacher." This teacher provides semantic guidance to a "student" encoder-decoder module. The student is tasked with generating segmentation maps of human body parts from the video frames. This process distills high-level semantic knowledge into the representation, forcing the model to prioritize human actors and their movements over irrelevant background noise.

2. Task-Dependent Multi-Task Learning

To resolve the issue of task interference, the framework employs a task-dependent architecture. Rather than forcing all tasks to share a single high-level feature space, the model uses a shared low-level encoder followed by distinct, specialized branches:

  • Knowledge Distillation Branch: Focuses on the semantic parsing of human body parts.
  • Contrastive Learning Branch: Focuses on motion contrast (distinguishing playback speeds) and appearance contrast (recognizing content across augmentations).

By using separate high-level encoders for these tasks, the model can specialize in both semantic depth and temporal dynamics simultaneously. The final video representation is a concatenation of these task-specific embeddings, providing a more comprehensive feature set for downstream action recognition.

Experimental Results

The model was pre-trained on the Kinetics-400 dataset and evaluated on two primary benchmarks: UCF101 and HMDB51. Using standard 3D architectures like C3D and R(2+1)D, the method demonstrated significant improvements:

  • State-of-the-Art Performance: On the UCF101 dataset, the model achieved an accuracy of 81.6% (using the R(2+1)D backbone), surpassing previous leading methods such as RSPNet.
  • Ablation Studies: The authors confirmed that both the human-centric prior and the task-dependent structure are vital. Removing the knowledge distillation module or reverting to a task-independent (shared) representation led to a measurable decline in accuracy.
  • Attention Visualization: Class Activation Maps (CAM) revealed that the proposed model's attention is more accurately centered on the human subjects compared to baseline SSL methods, which were frequently distracted by background elements.

Conclusion

By shifting the focus of self-supervised learning toward human-centered priors and addressing the inherent conflicts in multi-task learning, this research provides a more effective path for action recognition pre-training. The results suggest that injecting domain-specific semantic knowledge—specifically regarding human anatomy and movement—is key to moving beyond generic video representations toward more intelligent, task-aware models.

Summary updated

Abstract

Recently, much progress has been made for self-supervised action recognition. Most existing approaches emphasize the contrastive relations among videos, including appearance and motion consistency. However, two main issues remain for existing pre-training methods: 1) the learned representation is neutral and not informative for a specific task; 2) multi-task learning-based pre-training sometimes leads to sub-optimal solutions due to inconsistent domains of different tasks. To address the above issues, we propose a novel action recognition pre-training framework, which exploits human-centered prior knowledge that generates more informative representation, and avoids the conflict between multiple tasks by using task-dependent representations. Specifically, we distill knowledge from a human parsing model to enrich the semantic capability of representation. In addition, we combine knowledge distillation with contrastive learning to constitute a task-dependent multi-task framework. We achieve state-of-the-art performance on two popular benchmarks for action recognition task, i.e., UCF101 and HMDB51, verifying the effectiveness of our method.
BibTeX
@inproceedings{wang2022human,
  author = {Guanhong Wang and Keyu Lu and Yang Zhou and Zhanhao He and Gaoang Wang},
  title = {{Human-Centered Prior-Guided and Task-Dependent Multi-Task Representation Learning for Action Recognition Pre-Training}},
  booktitle = {ICME},
  year = {2022},
  url = {https://openreview.net/forum?id=GN3ASKjo3J},
}

Loading related content...

Figure