Abstract
Unsupervised domain adaptation (UDA) aims to adapt models trained on a labeled source domain to a completely unlabeled target domain. While many UDA techniques have been proposed for images, the problem remains largely underexplored for videos. We introduce Contrast and Mix (CoMix), a new contrastive learning framework for learning discriminative invariant feature representations for unsupervised video domain adaptation. CoMix utilizes temporal contrastive learning to bridge the domain gap by maximizing similarity between encoded representations of an unlabeled video at different speeds. Furthermore, we propose background mixing to leverage action semantics shared across domains and integrate supervised contrastive learning using target pseudo-labels. Extensive experiments demonstrate that CoMix achieves state-of-the-art performance on several benchmark datasets.
Introduction
Unsupervised Domain Adaptation (UDA) is critical for deploying models in real-world scenarios where labeled data is scarce in the target environment. In video action recognition, UDA faces unique challenges: temporal dynamics (actions occurring at varying speeds) and background bias (models overfitting to environmental cues rather than the action itself).
Existing video UDA methods often rely on adversarial learning to align features between source and target domains. However, adversarial training is notoriously unstable and difficult to optimize. CoMix addresses these issues by moving toward a more stable contrastive learning paradigm that explicitly handles temporal variations and background shifts.
Methodology
The CoMix framework is built upon three primary components designed to learn domain-invariant features.
Temporal Contrastive Learning (TCL)
TCL encourages the model to recognize that an action remains the same regardless of its temporal scale. By maximizing the similarity between representations of a video clip sampled at different rates (e.g., 16-clips vs. 8-clips) and minimizing similarity with other videos, the model learns features invariant to playback speed.
Background Mixing (BGM)
To prevent the model from "cheating" by relying on background information, CoMix introduces Background Mixing. Static backgrounds are extracted from one domain and blended with action sequences from another. These synthetic "mixed" videos serve as additional positive samples in the contrastive loss, forcing the network to focus on the foreground motion rather than the environmental context.
Target Pseudo-Labels (TPL)
To further refine the latent space, CoMix employs pseudo-labeling on the target domain. High-confidence predictions are used as labels for a supervised contrastive loss, which clusters videos of the same predicted class together, enhancing the discriminability of the learned features.
Experiments and Results
CoMix was evaluated on three major benchmarks: UCF-HMDB, Jester, and Epic-Kitchens.
Performance Benchmarks
- Jester: CoMix achieved a significant 9.2% improvement over the previous state-of-the-art (TA³N), reaching 64.7% accuracy.
- UCF-HMDB: The model reached an average accuracy of 90.3%, outperforming the SAVA method by 3.6%.
- Epic-Kitchens: CoMix demonstrated robustness in complex egocentric environments, achieving top performance in 5 out of 6 transfer tasks.
Ablation Study
Ablation results highlight the importance of each component. While TCL provides a strong baseline for temporal alignment, the addition of Background Mixing (BGM) consistently yields performance gains (e.g., +2.9% on UCF-HMDB), confirming that environmental invariance is a key driver of successful domain adaptation.
Conclusion
CoMix presents a robust alternative to adversarial domain adaptation for videos. By leveraging temporal speed invariance and explicit background mixing, the framework learns to prioritize core action semantics over domain-specific "nuisance variables." Future work may explore more sophisticated background extraction techniques and the application of BGM as a general-purpose data augmentation strategy for standard action recognition tasks.
Summary updated