Multi-Level Attentive Adversarial Learning With Temporal Dilation for Unsupervised Video Domain Adaptation

Peipeng Chen, Yuan Gao, Andy J. MaWACV 2022

TL;DR

This paper introduces MA2L-TD, a novel framework for unsupervised video domain adaptation that addresses limitations in existing methods by performing multi-level temporal feature alignment. By employing attentive adversarial learning across multiple temporal levels and a temporally-dilated aggregation module, MA2L-TD achieves state-of-the-art performance on challenging cross-domain action recognition tasks.

What Problem Does This Paper Address?

Video analysis, particularly action recognition, has seen significant advancements thanks to deep learning. However, training these models requires large, meticulously annotated datasets, which are expensive and time-consuming to create. Unsupervised Domain Adaptation (UDA) offers a solution by leveraging labeled data from a "source" domain to train models that can perform well on unlabeled data from a different "target" domain.

Existing UDA methods for video often struggle with the inherent complexity of video data. They typically align features at only two levels: individual frames and the entire video. This can lead to insufficient alignment, especially when dealing with variations in background, illumination, or camera pose. Furthermore, aligning only frame-level features with a single domain discriminator can mistakenly treat temporal variations within a video as domain differences, leading to misalignments. This paper tackles these issues by proposing a more sophisticated multi-level alignment strategy that explicitly considers temporal relationships.

Methods

The core of the proposed method, MA2L-TD (Multi-level Attentive Adversarial Learning with Temporal Dilation), lies in its multi-level approach to domain adaptation.

  • Multi-Level Attentive Adversarial Learning: Instead of a single domain discriminator, MA2L-TD employs multiple discriminators, each trained for a specific temporal level (e.g., frame, segment, video). This allows for more granular alignment. Crucially, an attention mechanism is introduced to weigh the importance of each level. These attention weights are dynamically determined by the degree of domain confusion at each level, meaning levels with larger domain gaps receive less attention. This prevents misalignments in one level from unduly influencing the overall adaptation.
  • Attentive Temporally-Dilated Aggregation: To generate multi-level temporal features efficiently, a novel aggregation module is proposed. This module uses temporal dilation, a technique that allows the model to consider features from non-consecutive time steps. This significantly reduces computational complexity compared to dense aggregation methods. The "attentive" aspect comes from assigning weights to features within the aggregation process, further mitigating the impact of potentially misaligned segments. This module balances the need for capturing rich temporal information with computational efficiency.
  • Task-Driven Feature Extractor: A general-purpose feature extractor (e.g., ResNet101) is used, and a task-driven feature extractor (MLP) is trained to adapt these features for the specific action recognition task. This extractor feeds into both the classification and adaptation branches.

Datasets

The proposed method is evaluated on four benchmark datasets:

  • UCF Olympic: A smaller dataset with 6 common classes shared between UCF101 and Olympic Sports.
  • UCF101 – HMDB51small: An intersection subset of UCF101 and HMDB51 with 5 classes.
  • UCF101 – HMDB51 full: A larger intersection subset with 12 classes.
  • Kinetics – Gameplay: A large-scale dataset with 30 classes.

The paper does not introduce new datasets but uses existing ones to demonstrate the effectiveness of MA2L-TD in various cross-domain scenarios.

Key Insights and Ideas

  • Multi-level alignment is crucial for complex video data: Simply aligning at frame and video levels is insufficient. Capturing intermediate temporal structures (segments) is vital for robust domain adaptation.
  • Attention for domain adaptation is powerful: Dynamically weighting different temporal levels based on domain confusion allows the model to focus on more reliable alignment signals and ignore noisy ones.
  • Temporal dilation offers an efficient way to capture long-range dependencies: This technique is key to building multi-level temporal features without incurring prohibitive computational costs.
  • Individual discriminators per level are better than shared ones: This allows for more precise alignment at each temporal granularity, preventing temporal variations from being mistaken for domain shifts.

Key Results

MA2L-TD consistently outperforms state-of-the-art methods across all tested benchmark datasets.

  • On the UCF – Olympic task, MA2L-TD achieves 100% accuracy, a significant improvement over the previous best of 98.15%.
  • On the UCF101 – HMDB51 full task, MA2L-TD achieves 85.00% accuracy, a substantial margin of 5.81% over the next best method.
  • Ablation studies demonstrate the effectiveness of each component: removing individual discriminators, multi-level alignment, level-wise attention, or the attentive temporally-dilated aggregation module all lead to performance degradation, validating the design choices.
  • t-SNE visualizations show that MA2L-TD effectively clusters source and target domain features of the same class together, while separating features of different classes, indicating better domain-invariant feature learning.

Limitations and Future Work

  • Parameter Sensitivity: The paper investigates the sensitivity to kernel size and level number, showing that more levels generally lead to higher accuracy. However, finding the optimal balance for different datasets might require further tuning.
  • Computational Cost: While temporal dilation improves efficiency, the multi-level nature and multiple discriminators still contribute to computational overhead.
  • Future Work: The authors suggest that further research could explore more advanced attention mechanisms or different aggregation strategies. The current work focuses on action recognition, and its applicability to other video understanding tasks could be explored.

Summary updated

BibTeX
@inproceedings{chen2022multi,
  author = {Peipeng Chen and Yuan Gao and Andy J. Ma},
  title = {{Multi-Level Attentive Adversarial Learning With Temporal Dilation for Unsupervised Video Domain Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure