Human-Centric Transformer for Domain Adaptive Action Recognition

Kun-Yu Lin, Jiaming Zhou, Wei-Shi Zheng

TL;DR

The HCTransformer addresses the "domain gap" in action recognition by explicitly decoupling video features into human-centric and context-centric cues. By focusing on the person and their direct interactions rather than the background, the model achieves state-of-the-art performance in transferring recognition power from labeled source domains to unlabeled target domains.

What Problem Does This Paper Address?

Action recognition models often suffer a massive performance drop when deployed in the real world. A model trained to recognize "shooting a basketball" in a well-lit indoor gym might fail completely when it sees the same action performed on a sunny outdoor court.

This happens because standard models often rely on spurious correlations—they learn to recognize the "gym" rather than the "shooting motion." In technical terms, this is a domain adaptation problem. While previous methods tried to align global video features across domains, they often lost "human cues" in the process, focusing instead on non-human contexts that don't generalize. The authors argue that since humans perform the actions, we must prioritize human-centric information to bridge the domain gap.

Methods

The authors introduce the Human-Centric Transformer (HCTransformer), which uses a decoupled learning paradigm to isolate and align three specific types of information:

  1. Spatial Masking Module: Using an off-the-shelf human detector, the model separates pixels into "human" and "context" sets for every video clip.
  2. Human Encoder: This branch focuses purely on the temporal dynamics of the person. It uses a Temporal Relation Network (TRN) to model how a person moves across different time scales.
  3. Context Encoder: Instead of looking at the whole background, this module uses "context prototypes" (learned via K-means clustering) to identify objects or scenes that are actually correlated with the action (like a bow in archery) while ignoring irrelevant background noise.
  4. Human-Context Decoder: A Transformer-based architecture that uses cross-attention to model the interaction between the human and the action-correlated context.
  5. Multi-level Domain Alignment: The model applies adversarial alignment losses (following the DANN framework) at every stage—human features, context features, and interaction features—to ensure they are domain-invariant.

Datasets

The model was evaluated on three major benchmarks for domain adaptive action recognition:

  • UCF-HMDB: The standard benchmark covering 12 overlapping categories between the UCF101 and HMDB51 datasets.
  • Kinetics-NecDrone: A challenging "large-gap" dataset where the source is YouTube videos (Kinetics) and the target is drone-captured footage (NecDrone).
  • EPIC-Kitchens-UDA: A large-scale egocentric (first-person) dataset focused on kitchen activities, used here to test "hand-centric" adaptation.

Key Insights and Ideas

  • Context is a Double-Edged Sword: While background context helps recognition within a single dataset, "action-agnostic" context (like the floor of a fencing court) leads to errors when the environment changes.
  • Decoupling over Fusion: Simply "fusing" human and context features (late fusion) provides minimal gains. The "aha moment" here is that explicitly decoupling them during the feature extraction process allows the model to align specific human-centric cues that are more robust to domain shifts.
  • Robustness to "Messy" Masks: Interestingly, the model does not require perfect human segmentation. Even rough bounding boxes from off-the-shelf detectors are enough to guide the Transformer to the right regions.
  • Joint Information: Quantitative analysis showed that HCTransformer encodes significantly more "joint" (static + dynamic) information compared to previous models, which explains its superior temporal modeling.

Key Results

The HCTransformer achieved state-of-the-art results across the board:

  • UCF-HMDB: Improved accuracy by up to 6.9% on the UCF $\rightarrow$ HMDB task compared to the previous leading method (TA3N).
  • Kinetics-NecDrone: Achieved a 5.4% improvement in accuracy, demonstrating high effectiveness even when the camera viewpoint changes drastically (e.g., from eye-level to drone-view).
  • Qualitative Success: Grad-CAM visualizations show that while older models (like TA3N) often stare at the ground or walls, HCTransformer consistently "looks" at the person and the objects they are interacting with (e.g., the ball, the bow, or the golf club).

Limitations and Future Work

  • Detector Dependency: While robust, the model still relies on an external human/hand detector. If the detector fails completely (e.g., extremely low light or heavy occlusion), the spatial masking becomes less effective.
  • Modality Limits: This paper focuses on RGB video. The authors suggest that future work could incorporate other modalities like audio or depth information.
  • Subject Diversity: The current work focuses on human-centric cues; the authors note that exploring "action subject changes" (e.g., adapting from humans to robots performing similar tasks) could be a promising future direction.

Summary updated

Abstract

We study the domain adaptation task for action recognition, namely domain adaptive action recognition, which aims to effectively transfer action recognition power from a label-sufficient source domain to a label-free target domain. Since actions are performed by humans, it is crucial to exploit human cues in videos when recognizing actions across domains. However, existing methods are prone to losing human cues but prefer to exploit the correlation between non-human contexts and associated actions for recognition, and the contexts of interest agnostic to actions would reduce recognition performance in the target domain. To overcome this problem, we focus on uncovering human-centric action cues for domain adaptive action recognition, and our conception is to investigate two aspects of human-centric action cues, namely human cues and human-context interaction cues. Accordingly, our proposed Human-Centric Transformer (HCTransformer) develops a decoupled human-centric learning paradigm to explicitly concentrate on human-centric action cues in domain-variant video feature learning. Our HCTransformer first conducts human-aware temporal modeling by a human encoder, aiming to avoid a loss of human cues during domain-invariant video feature learning. Then, by a Transformer-like architecture, HCTransformer exploits domain-invariant and action-correlated contexts by a context encoder, and further models domain-invariant interaction between humans and action-correlated contexts. We conduct extensive experiments on three benchmarks, namely UCF-HMDB, Kinetics-NecDrone and EPIC-Kitchens-UDA, and the state-of-the-art performance demonstrates the effectiveness of our proposed HCTransformer.
BibTeX
@article{lin2025human,
  author = {Kun-Yu Lin and Jiaming Zhou and Wei-Shi Zheng},
  title = {{Human-Centric Transformer for Domain Adaptive Action Recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  url = {https://openreview.net/forum?id=eW36cYDS53},
}
Figure