Abstract
Understanding temporal dynamics of video is an essential aspect of learning better video representations. Recently, transformer-based architectural designs have been extensively explored for video tasks due to their capability to capture long-term dependency of input sequences. However, we found that these Video Transformers are still biased to learn spatial dynamics rather than temporal ones, and debiasing the spurious correlation is critical for their performance. Based on the observations, we design simple yet effective self-supervised tasks for video models to learn temporal dynamics better. Specifically, for debiasing the spatial bias, our method learns the temporal order of video frames as extra self-supervision and enforces the randomly shuffled frames to have low-confidence outputs. Also, our method learns the temporal flow direction of video tokens among consecutive frames for enhancing the correlation toward temporal dynamics. Under various video action recognition tasks, we demonstrate the effectiveness of our method and its compatibility with state-of-the-art Video Transformers.
Introduction
Video Transformers have emerged as powerful architectures for video tasks, leveraging their ability to capture long-term dependencies. However, a critical observation in recent research is that these models often exhibit a strong bias toward spatial dynamics—identifying static objects rather than understanding movement. This paper introduces TIME (Time Is MattEr), a model-agnostic framework designed to enhance temporal reasoning through three targeted self-supervised tasks.
The Problem: Spatial Bias and Vanishing Time
Standard Video Transformers often "cheat" by relying on static features. The authors demonstrate this through a "shuffling test": if a model truly understands temporal dynamics, its performance should degrade significantly when video frames are randomly reordered. Surprisingly, current models like TimeSformer maintain high accuracy on shuffled videos, suggesting they treat videos as a "bag of images" rather than a sequential narrative.
Furthermore, the authors identify a phenomenon they call "vanishing time." Empirical evidence shows that the ability to predict the temporal order of frames decreases significantly as data progresses through the Transformer's layers. This suggests that temporal resolution is lost in deeper blocks, which are supposed to be capturing high-level semantics.
The TIME Framework
To address these issues, the authors propose three self-supervised loss functions that can be integrated into existing architectures to force a focus on temporal dynamics:
- Debiasing Spatial Dynamics ($L_{debias}$): The model is trained on shuffled videos and forced to produce low-confidence (uniform) outputs. This penalizes the model for making predictions based solely on static objects found in individual frames.
- Learning Temporal Order ($L_{order}$): A classifier is added to the final block to predict the original absolute position of each frame, forcing the model to preserve temporal information throughout its depth.
- Learning Temporal Flow ($L_{flow}$): This token-level task forces the model to predict the direction of motion (one of eight angular directions or "center") for specific patches between consecutive frames. By utilizing a classical computer vision algorithm (Farnebäck’s) to generate these "flow labels" automatically, the model learns to correlate local patch movements with global action labels.
Experimental Results
The framework was evaluated primarily on Something-Something-v2 (SSv2), a dataset requiring strong temporal reasoning. TIME improved TimeSformer accuracy by 1.6% and X-ViT by 3.4%. On the "static-biased" Kinetics-400 dataset, gains were more marginal, reflecting the dataset's inherent nature where spatial cues often suffice for classification.
A key insight is the "Confidence Gap"—the difference in model confidence between original and shuffled videos. TIME successfully widens this gap, indicating a genuine shift toward temporal understanding. Additionally, the method showed surprising cross-domain utility; when applied to image classification (DeiT-Ti), it improved robustness against background shifts on the ImageNet-9 benchmark by 8.6%, demonstrating that the debiasing logic helps models focus on foreground objects by ignoring background clutter.
Conclusion
TIME provides a simple yet effective suite of self-supervised tasks to debias Video Transformers from spatial cues and enhance their temporal reasoning capabilities. By forcing models to respect temporal order and motion flow, the framework achieves significant performance gains on action recognition benchmarks and improves robustness in image classification. Future work may explore architectural modifications to further integrate temporal flow at a structural level.
Summary updated