Improving Video Model Transfer with Dynamic Representation Learning

Yi Li, Nuno Vasconcelos

Abstract

Current video recognition models often rely on static spatial cues rather than motion, leading to high benchmark scores but poor generalization. We introduce Dynamic Representation Learning (DRL), an adversarial framework that forces models to prioritize temporal dynamics. By maximizing a "dynamic score" through a min-max game between a video teacher and a spatial student, DRL significantly improves model transferability to unseen domains and motion-heavy tasks.

Introduction

Deep learning has advanced video recognition, yet many models suffer from "spatial bias." They often identify actions by detecting static objects or backgrounds (e.g., identifying "horseback riding" solely by the presence of a horse) rather than actual movement. While these shortcuts boost performance on biased datasets, they fail in new environments where such cues are absent or misleading. This paper addresses this lack of true temporal understanding by proposing a method to measure and actively encourage the learning of dynamic representations.

Methods: Dynamic Representation Learning (DRL)

The authors propose a two-pronged approach: a metric to quantify temporal modeling and an adversarial training strategy to enhance it.

Dynamic Scoring

The Dynamic Score measures the unique information a video model captures beyond what a purely spatial model can. It is calculated by training a "spatial student" (a 2D network) to mimic a "video teacher." A low dynamic score indicates high spatial bias, as the student can easily replicate the teacher's predictions using only static frames.

Adversarial Training

Dynamic Representation Learning (DRL) implements a min-max game: the student attempts to minimize the prediction gap with the teacher, while the teacher (the video model) attempts to maximize this gap (the dynamic score) while maintaining classification accuracy.

The DRL framework consists of Dynamic Scoring to measure spatial bias and Representation Learning to adversarially train the video model.

Implementation strategies include:

  • Adversarial Augmentation: Using noise to obscure spatial cues.
  • Direct Optimization: A formal min-max optimization between networks.
  • Fast Approximation: Using "frozen" (repeated) or "shuffled" frames to strip temporal info efficiently.

Experiments and Results

The framework was evaluated across diverse benchmarks, including Kinetics-400, UCF-101, HMDB-51, and motion-critical datasets like Diving-48 and Mimetics.

The "Cheating" Trade-off

DRL models often show a slight performance drop (2-3%) on the original Kinetics test set. This occurs because the model is forced to ignore the "easy" spatial shortcuts it previously used for high scores. However, this trade-off results in superior robustness during transfer.

Transfer and Generalization

DRL models significantly outperform baselines when moved to new datasets. On the Mimetics dataset, which features actions without typical contexts, DRL achieved 26.4% accuracy compared to the baseline's 18.9%. In few-shot gesture recognition on Jester, DRL improved performance by up to 8%.

Qualitative comparison showing how DRL corrects spatial bias. Baselines often misclassify actions based on background objects (e.g., "Deadlifting" for a surfer in a gym), while DRL focuses on motion.

Key Insights

  • Freezing vs. Shuffling: "Freezing" a clip (repeating one frame) proved more effective than shuffling. Shuffling introduces artifacts, whereas freezing provides a clean static baseline.
  • Pervasiveness of Bias: Even multi-modal architectures like two-stream networks exhibit spatial bias in their RGB branches, highlighting the broad utility of DRL.

Conclusion

Dynamic Representation Learning effectively mitigates spatial bias in video models. By prioritizing temporal dynamics over static shortcuts, DRL produces representations that generalize far better to unseen domains. While the computational overhead of adversarial training remains a challenge, the gains in transferability and true motion understanding represent a significant step forward for robust video recognition.

Summary updated

Abstract

Temporal modeling is an essential element in video understanding. While deep convolution-based architectures have been successful at solving large-scale video recognition datasets, recent work has pointed out that they are biased towards modeling short-range relations, often failing to capture long-term temporal structures in the videos, leading to poor transfer and generalization to new datasets. In this work, the problem of dynamic representation learning (DRL) is studied. We propose dynamic score, a measure of video dynamic modeling that describes the additional amount of information learned by a video network that cannot be captured by pure spatial student through knowledge distillation. DRL is then formulated as an adversarial learning problem between the video and spatial models, with the objective of maximizing the dynamic score of learned spatiotemporal classifier. The quality of learned video representations are evaluated on a diverse set of transfer learning problems concerning many-shot and few-shot action classification. Experimental results show that models learned with DRL outperform baselines in dynamic modeling, demonstrating higher transferability and generalization capacity to novel domains and tasks.
BibTeX
@inproceedings{li2022improving,
  author = {Yi Li and Nuno Vasconcelos},
  title = {{Improving Video Model Transfer with Dynamic Representation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01868},
}
Figure