Memory Efficient Temporal & Visual Graph Model for Unsupervised Video Domain Adaptation

Xinyue Hu, Lin Gu, Liangchen Liu, Ruijiang Li, Chang Su, Tatsuya Harada, Yingying ZhuarXiv 2022 · OpenReview

Abstract

Existing video domain adaptation (DA) methods need to store all temporal combinations of video frames or pair the source and target videos, which are memory cost expensive and can't scale up to long videos. To address these limitations, we propose a memory-efficient graph-based video DA approach. Our method models each source or target video as a graph where nodes represent video frames and edges represent the temporal or visual similarity relationship between frames. We use a graph attention network to learn the weight of individual frames and simultaneously align the source and target video into a domain-invariant graph feature space. Instead of storing a large number of sub-videos, our method only constructs one graph with a graph attention mechanism for one video, reducing the memory cost substantially. The extensive experiments show that, compared with the state-of-art methods, we achieved superior performance while reducing the memory cost significantly.

Introduction

Unsupervised Domain Adaptation (UDA) in video analysis is critical for deploying models across varying environments, such as recognizing actions in different lighting or backgrounds. However, video UDA is particularly challenging due to shifts in both spatial (image-level) and temporal (action speed/duration) domains. Current state-of-the-art methods often hit a "memory wall." They typically rely on storing exhaustive temporal combinations of frames or solving NP-hard pairing problems between source and target domains, leading to exponential memory growth as video length increases.

Methodology

The proposed approach addresses these bottlenecks by representing each video as a single, unified graph. This shift from quadratic to linear complexity ($O(nv)$ space) is the primary driver for memory efficiency.

Graph Construction

Each video is modeled with:

  • Nodes: Representing individual frames with features extracted from backbones like ResNet-101 or I3D.
  • Temporal Edges: Connecting adjacent frames to capture short-term motion dynamics.
  • Visual Similarity Edges: Connecting frames with high feature similarity, regardless of temporal distance, to capture long-term dependencies.

Architecture and Alignment

The model employs a 3-layer Graph Attention Network (GAT) to process these graphs. The attention mechanism dynamically weights the importance of frames, focusing on core actions while ignoring irrelevant segments. Pooling layers then compress the graph into a compact, domain-invariant representation.

The overall architecture of the proposed graph-based video domain adaptation model, featuring the Graph Module, GAT layers, and domain discriminators.

As shown in the architecture diagram, the system integrates domain discriminators ($D_f$ and $D_v$) and gradient reversal layers to ensure the learned features are indistinguishable between the source and target domains, facilitating effective adaptation.

Experimental Results

The model was evaluated on standard benchmarks including UCF-HMDB (small and full) and UCF-Olympic. These datasets test the model's ability to transfer knowledge between different video distributions (e.g., UCF101 vs. HMDB51).

Performance and Efficiency

The proposed method achieved state-of-the-art accuracy, notably reaching 98.15% on the UCF-Olympic (U→O) task. Crucially, it maintains this performance while being significantly more memory-efficient than previous methods like TA$^3$N and ABG.

Comparison of GPU memory usage vs. number of input frames. The proposed method exhibits linear memory growth, staying well below the GPU maximum memory limit compared to baseline methods.

As illustrated in the memory analysis, the graph-based approach achieves approximately a 65% reduction in GPU memory usage for a 25-frame input. While competing methods show exponential or steep memory growth that quickly hits hardware limits, our model scales linearly, enabling the processing of much longer video sequences.

Conclusion

By shifting from exhaustive sub-video combinations to a single graph representation with visual and temporal edges, this work provides a scalable solution for video UDA. The use of graph attention allows for effective feature weighting and domain alignment without the prohibitive memory costs of prior art. Future work will explore integrating more granular spatial attention to further improve robustness against complex background shifts.

Summary updated

Abstract

Existing video domain adaption (DA) methods need to store all temporal combinations of video frames or pair the source and target videos, which are memory cost expensive and can't scale up to long videos. To address these limitations, we propose a memory-efficient graph-based video DA approach as follows. At first our method models each source or target video by a graph: nodes represent video frames and edges represent the temporal or visual similarity relationship between frames. We use a graph attention network to learn the weight of individual frames and simultaneously align the source and target video into a domain-invariant graph feature space. Instead of storing a large number of sub-videos, our method only constructs one graph with a graph attention mechanism for one video, reducing the memory cost substantially. The extensive experiments show that, compared with the state-of-art methods, we achieved superior performance while reducing the memory cost significantly.
BibTeX
@misc{hu2022memory,
  author = {Xinyue Hu and Lin Gu and Liangchen Liu and Ruijiang Li and Chang Su and Tatsuya Harada and Yingying Zhu},
  title = {{Memory Efficient Temporal \& Visual Graph Model for Unsupervised Video Domain Adaptation}},
  year = {2022},
  eprint = {2208.06554},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.06554},
}
Figure