Abstract
Current video recognition models often rely on static spatial cues rather than motion, leading to high benchmark scores but poor generalization. We introduce Dynamic Representation Learning (DRL), an adversarial framework that forces models to prioritize temporal dynamics. By maximizing a "dynamic score" through a min-max game between a video teacher and a spatial student, DRL significantly improves model transferability to unseen domains and motion-heavy tasks.
Introduction
Deep learning has advanced video recognition, yet many models suffer from "spatial bias." They often identify actions by detecting static objects or backgrounds (e.g., identifying "horseback riding" solely by the presence of a horse) rather than actual movement. While these shortcuts boost performance on biased datasets, they fail in new environments where such cues are absent or misleading. This paper addresses this lack of true temporal understanding by proposing a method to measure and actively encourage the learning of dynamic representations.
Methods: Dynamic Representation Learning (DRL)
The authors propose a two-pronged approach: a metric to quantify temporal modeling and an adversarial training strategy to enhance it.
Dynamic Scoring
The Dynamic Score measures the unique information a video model captures beyond what a purely spatial model can. It is calculated by training a "spatial student" (a 2D network) to mimic a "video teacher." A low dynamic score indicates high spatial bias, as the student can easily replicate the teacher's predictions using only static frames.
Adversarial Training
Dynamic Representation Learning (DRL) implements a min-max game: the student attempts to minimize the prediction gap with the teacher, while the teacher (the video model) attempts to maximize this gap (the dynamic score) while maintaining classification accuracy.
Implementation strategies include:
- Adversarial Augmentation: Using noise to obscure spatial cues.
- Direct Optimization: A formal min-max optimization between networks.
- Fast Approximation: Using "frozen" (repeated) or "shuffled" frames to strip temporal info efficiently.
Experiments and Results
The framework was evaluated across diverse benchmarks, including Kinetics-400, UCF-101, HMDB-51, and motion-critical datasets like Diving-48 and Mimetics.
The "Cheating" Trade-off
DRL models often show a slight performance drop (2-3%) on the original Kinetics test set. This occurs because the model is forced to ignore the "easy" spatial shortcuts it previously used for high scores. However, this trade-off results in superior robustness during transfer.
Transfer and Generalization
DRL models significantly outperform baselines when moved to new datasets. On the Mimetics dataset, which features actions without typical contexts, DRL achieved 26.4% accuracy compared to the baseline's 18.9%. In few-shot gesture recognition on Jester, DRL improved performance by up to 8%.
Key Insights
- Freezing vs. Shuffling: "Freezing" a clip (repeating one frame) proved more effective than shuffling. Shuffling introduces artifacts, whereas freezing provides a clean static baseline.
- Pervasiveness of Bias: Even multi-modal architectures like two-stream networks exhibit spatial bias in their RGB branches, highlighting the broad utility of DRL.
Conclusion
Dynamic Representation Learning effectively mitigates spatial bias in video models. By prioritizing temporal dynamics over static shortcuts, DRL produces representations that generalize far better to unseen domains. While the computational overhead of adversarial training remains a challenge, the gains in transferability and true motion understanding represent a significant step forward for robust video recognition.
Summary updated