Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing

Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, Abir Das

Abstract

Unsupervised domain adaptation (UDA) aims to adapt models trained on a labeled source domain to a completely unlabeled target domain. While many UDA techniques have been proposed for images, the problem remains largely underexplored for videos. We introduce Contrast and Mix (CoMix), a new contrastive learning framework for learning discriminative invariant feature representations for unsupervised video domain adaptation. CoMix utilizes temporal contrastive learning to bridge the domain gap by maximizing similarity between encoded representations of an unlabeled video at different speeds. Furthermore, we propose background mixing to leverage action semantics shared across domains and integrate supervised contrastive learning using target pseudo-labels. Extensive experiments demonstrate that CoMix achieves state-of-the-art performance on several benchmark datasets.

Introduction

Unsupervised Domain Adaptation (UDA) is critical for deploying models in real-world scenarios where labeled data is scarce in the target environment. In video action recognition, UDA faces unique challenges: temporal dynamics (actions occurring at varying speeds) and background bias (models overfitting to environmental cues rather than the action itself).

Existing video UDA methods often rely on adversarial learning to align features between source and target domains. However, adversarial training is notoriously unstable and difficult to optimize. CoMix addresses these issues by moving toward a more stable contrastive learning paradigm that explicitly handles temporal variations and background shifts.

Methodology

The CoMix framework is built upon three primary components designed to learn domain-invariant features.

Temporal Contrastive Learning (TCL)

TCL encourages the model to recognize that an action remains the same regardless of its temporal scale. By maximizing the similarity between representations of a video clip sampled at different rates (e.g., 16-clips vs. 8-clips) and minimizing similarity with other videos, the model learns features invariant to playback speed.

Temporal Contrastive Learning (TCL) maximizes similarity between different temporal augmentations of the same video.

Background Mixing (BGM)

To prevent the model from "cheating" by relying on background information, CoMix introduces Background Mixing. Static backgrounds are extracted from one domain and blended with action sequences from another. These synthetic "mixed" videos serve as additional positive samples in the contrastive loss, forcing the network to focus on the foreground motion rather than the environmental context.

Background Mixing (BGM) provides additional positive samples by blending actions with backgrounds from the opposite domain.

Target Pseudo-Labels (TPL)

To further refine the latent space, CoMix employs pseudo-labeling on the target domain. High-confidence predictions are used as labels for a supervised contrastive loss, which clusters videos of the same predicted class together, enhancing the discriminability of the learned features.

Experiments and Results

CoMix was evaluated on three major benchmarks: UCF-HMDB, Jester, and Epic-Kitchens.

Performance Benchmarks

  • Jester: CoMix achieved a significant 9.2% improvement over the previous state-of-the-art (TA³N), reaching 64.7% accuracy.
  • UCF-HMDB: The model reached an average accuracy of 90.3%, outperforming the SAVA method by 3.6%.
  • Epic-Kitchens: CoMix demonstrated robustness in complex egocentric environments, achieving top performance in 5 out of 6 transfer tasks.

Ablation Study

Ablation results highlight the importance of each component. While TCL provides a strong baseline for temporal alignment, the addition of Background Mixing (BGM) consistently yields performance gains (e.g., +2.9% on UCF-HMDB), confirming that environmental invariance is a key driver of successful domain adaptation.

Conclusion

CoMix presents a robust alternative to adversarial domain adaptation for videos. By leveraging temporal speed invariance and explicit background mixing, the framework learns to prioritize core action semantics over domain-specific "nuisance variables." Future work may explore more sophisticated background extraction techniques and the application of BGM as a general-purpose data augmentation strategy for standard action recognition tasks.

Summary updated

Abstract

Unsupervised domain adaptation which aims to adapt models trained on a labeled source domain to a completely unlabeled target domain has attracted much attention in recent years. While many domain adaptation techniques have been proposed for images, the problem of unsupervised domain adaptation in videos remains largely underexplored. In this paper, we introduce Contrast and Mix (CoMix), a new contrastive learning framework that aims to learn discriminative invariant feature representations for unsupervised video domain adaptation. First, unlike existing methods that rely on adversarial learning for feature alignment, we utilize temporal contrastive learning to bridge the domain gap by maximizing the similarity between encoded representations of an unlabeled video at two different speeds as well as minimizing the similarity between different videos played at different speeds. Second, we propose a novel extension to the temporal contrastive loss by using background mixing that allows additional positives per anchor, thus adapting contrastive learning to leverage action semantics shared across both domains. Moreover, we also integrate a supervised contrastive learning objective using target pseudo-labels to enhance discriminability of the latent space for video domain adaptation. Extensive experiments on several benchmark datasets demonstrate the superiority of our proposed approach over state-of-the-art methods. Project page: https://cvir.github.io/projects/comix
BibTeX
@inproceedings{sahoo2021contrast,
  author = {Aadarsh Sahoo and Rutav Shah and Rameswar Panda and Kate Saenko and Abir Das},
  title = {{Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=a1wQOh27zcy},
}
Figure