Discovering Informative and Robust Positives for Video Domain Adaptation

Chang Liu, Kunpeng Li, Michael Stopa, Jun Amano, Yun FuICLR 2023 · OpenReview

Introduction

This paper introduces a non-contrastive learning framework for Video Domain Adaptation (VDA) that eliminates the need for negative samples, which often act as "false negatives" and hinder model performance. By mining "informative" intra-domain positives through augmentations and nearest neighbors, and "robust" cross-domain positives via synthesized features from estimated target distributions, the authors achieve state-of-the-art results on major action recognition benchmarks.

The Challenge of Video Domain Adaptation

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled "source" dataset (e.g., professional movies) to an unlabeled "target" dataset (e.g., low-quality YouTube clips). In video recognition, this is particularly hard because the "domain shift" occurs in both appearance (spatial) and motion (temporal).

Previous approaches relied heavily on contrastive learning, which pulls "positive" pairs (same class) together and pushes "negative" pairs (different classes) apart. However, the authors identify three major bottlenecks in this paradigm:

  1. Limited Intra-domain Variations: Standard augmentations aren't diverse enough to help the model generalize within the target domain.
  2. False Cross-domain Positives: Since target labels are unknown, models use "pseudo-labels" which are often wrong, leading the model to align the wrong classes across domains.
  3. False Negatives: Contrastive learning often accidentally pushes apart samples that actually belong to the same class but are labeled as negatives, destroying semantic clusters.

A Non-Contrastive Framework

The authors propose a non-contrastive framework that focuses entirely on "positives."

Proposed Non-Contrastive Framework

Their approach is built on three main pillars:

  • Informative Intra-domain Positives: To help the model understand the target domain better, they use an unlabeled target video as an "anchor." They then find "positives" in two ways: (1) heavy spatial and temporal augmentations (like changing play speed or shuffling clips) and (2) finding the video's Nearest Neighbors (NNs) in the feature space. They found that NNs in the target domain are surprisingly "pure" (correctly matched) compared to noisy pseudo-labels.
  • Robust Cross-domain Positives: To align source and target domains without being misled by noisy labels, they don't just pick a target video. Instead, they estimate a Gaussian distribution (mean and covariance) for each class in the target domain. They then "synthesize" target features from this distribution to serve as positives for source anchors. This averaging effect makes the alignment much more robust to individual labeling errors.
  • Non-Contrastive Optimization: Borrowing from architectures like SimSiam, they use a predictor MLP and a stop-gradient operation. This allows the model to learn from positive pairs without needing any negative samples at all, completely bypassing the "false negative" problem.

Key Insights and Ideas

Analysis of False Negatives and Purity
  • The "Purity" of Neighbors: One of the most interesting findings is that a target video's nearest neighbors in the feature space are more likely to be the same class than a pseudo-label generated by a classifier. This makes NNs a "cleaner" source of supervision.
  • Covariance Matters: While previous works used class centroids (means) for alignment, this paper shows that estimating the covariance (the "shape" of the class distribution) allows for synthesizing more diverse and realistic features, leading to better domain alignment.
  • The MLP Predictor Paradox: The authors discovered that while an MLP predictor head is crucial for optimizing intra-domain features (preventing collapse), it actually hampers convergence for cross-domain alignment. Consequently, they remove it for the cross-domain part of their loss function.

Datasets and Key Results

The model was evaluated on two challenging cross-domain action recognition benchmarks:

  • UCF $\leftrightarrow$ HMDB: A standard benchmark involving 12 overlapping classes between the UCF101 and HMDB51 datasets.
  • Epic-Kitchens: A fine-grained egocentric (first-person) dataset. The authors tested on six different transfer tasks between different kitchens (D1, D2, and D3), which feature significant imbalances and domain gaps.

The proposed method consistently outperformed previous state-of-the-art (SOTA) methods:

  • UCF-HMDB: Achieved 84.0% on UCF $\to$ HMDB and 94.7% on HMDB $\to$ UCF using an I3D backbone, representing a 1.8% to 3.5% improvement over previous contrastive methods.
  • Epic-Kitchens: Achieved a mean accuracy of 52.3% across six tasks, outperforming the previous best (LCMC) by 1.1%.
  • Ablation Studies: Removing the "non-contrastive" aspect (adding negatives back in) caused performance to drop by about 1.5%, proving that false negatives are indeed a significant headwind in VDA.

Limitations and Future Work

While the method is robust, the authors note that it currently focuses primarily on RGB and Flow modalities. They suggest that multimodal learning—specifically mining cross-modal nearest neighbors (e.g., using audio to find similar video clips)—is a promising next step. Additionally, the reliance on a memory bank to estimate distributions adds some computational overhead, though the authors derived a closed-form upper bound to make the cross-domain optimization more efficient.

Summary updated

Abstract

Unsupervised domain adaptation for video recognition is challenging where the domain shift includes both spatial variations and temporal dynamics. Previous works have focused on exploring contrastive learning for cross-domain alignment. However, limited variations in intra-domain positives, false cross-domain positives, and false negatives hinder contrastive learning from fulfilling intra-domain discrimination and cross-domain closeness. This paper presents a non-contrastive learning framework without relying on negative samples for unsupervised video domain adaptation. To address the limited variations in intra-domain positives, we set unlabeled target videos as anchors and explored to mine "informative intra-domain positives" in the form of spatial/temporal augmentations and target nearest neighbors (NNs). To tackle the false cross-domain positives led by noisy pseudo-labels, we reversely set source videos as anchors and sample the synthesized target videos as "robust cross-domain positives" from an estimated target distribution, which are naturally more robust to the pseudo-label noise. Our approach is demonstrated to be superior to state-of-the-art methods through extensive experiments on several cross-domain action recognition benchmarks.
BibTeX
@inproceedings{liu2023discovering,
  author = {Chang Liu and Kunpeng Li and Michael Stopa and Jun Amano and Yun Fu},
  title = {{Discovering Informative and Robust Positives for Video Domain Adaptation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=vk-j5pQY3Gv},
}
Figure