Human-Centric Transformer for Domain Adaptive Action Recognition

Kun-Yu Lin, Jiaming Zhou, Wei-Shi Zheng

Abstract

Action recognition models often struggle with domain shifts, where a model trained in one environment (e.g., indoors) fails in another (e.g., outdoors). This paper introduces the Human-Centric Transformer (HCTransformer), which addresses this by explicitly decoupling video features into human-centric and context-centric components. By focusing on human temporal dynamics and action-relevant context prototypes, HCTransformer achieves state-of-the-art results on domain adaptive action recognition benchmarks, effectively bridging the gap between labeled source domains and unlabeled target domains.

Introduction

Action recognition models frequently suffer from performance degradation when deployed in real-world scenarios different from their training data. For instance, a model trained to recognize "shooting a basketball" in a well-lit indoor gym might fail on a sunny outdoor court. This occurs because standard models often rely on spurious correlations, learning to recognize the background environment (the "gym") rather than the action itself (the "shooting motion"). In technical terms, this is a domain adaptation problem. While previous methods attempted to align global video features, they often lost critical human-centric cues. The HCTransformer prioritizes human information to ensure robust cross-domain generalization.

Methodology

The HCTransformer utilizes a decoupled learning paradigm to isolate and align three specific types of information: human dynamics, action-relevant context, and their interaction.

The HCTransformer architecture decouples human and context features through specialized encoders and a joint decoder.

Key Components:

  1. Spatial Masking Module: Employs an off-the-shelf human detector to separate pixels into "human" and "context" sets for every video clip.
  2. Human Encoder: Focuses purely on the temporal dynamics of the person using a Temporal Relation Network (TRN) to model movement across different time scales.
  3. Context Encoder: Identifies objects or scenes correlated with the action (e.g., a bow in archery) using "context prototypes" learned via K-means clustering, while ignoring irrelevant background noise.
  4. Human-Context Decoder: A Transformer-based architecture that uses cross-attention to model the interaction between the human and the action-correlated context.
  5. Multi-level Domain Alignment: Applies adversarial alignment losses at every stage—human, context, and interaction features—to ensure they are domain-invariant.

Experiments and Datasets

The model was evaluated on three major benchmarks for domain adaptive action recognition:

  • UCF-HMDB: The standard benchmark covering 12 overlapping categories between the UCF101 and HMDB51 datasets.
  • Kinetics-NecDrone: A challenging "large-gap" dataset where the source is YouTube videos and the target is drone-captured footage.
  • EPIC-Kitchens-UDA: A large-scale egocentric dataset used to test "hand-centric" adaptation in kitchen activities.

Results and Discussion

The HCTransformer achieved state-of-the-art results across all benchmarks. On the UCF $\rightarrow$ HMDB task, it improved accuracy by up to 6.9% compared to the previous leading method (TA3N). In the Kinetics-NecDrone task, it achieved a 5.4% improvement, demonstrating effectiveness even with drastic camera viewpoint changes.

Grad-CAM visualizations show HCTransformer (bottom) focusing on humans and relevant objects, whereas TA3N (top) often focuses on irrelevant background.

Qualitative analysis via Grad-CAM visualizations (see above) confirms that while older models often focus on the ground or walls, HCTransformer consistently attends to the person and the objects they interact with, such as the ball, bow, or golf club. This decoupling approach proves that context is a "double-edged sword": while it helps within a single dataset, action-agnostic context leads to errors when the environment changes.

Conclusion

The HCTransformer demonstrates that explicitly decoupling human-centric and context-centric cues is superior to simple feature fusion for domain adaptation. By prioritizing the subjects performing the actions and the objects they interact with, the model achieves significant robustness to environmental shifts. Future work may explore incorporating additional modalities like audio or depth information to further enhance recognition accuracy.

Summary updated

BibTeX
@article{lin2025human,
  author = {Kun-Yu Lin and Jiaming Zhou and Wei-Shi Zheng},
  title = {{Human-Centric Transformer for Domain Adaptive Action Recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  url = {https://openreview.net/forum?id=eW36cYDS53},
}
Figure