Introduction
Modern action recognition models, while achieving high accuracy on standard benchmarks, often suffer from a lack of generalization across different environments. Much like the "texture bias" observed in 2D image classification—where models rely on local textures rather than global shapes—video models frequently exploit spurious spatial correlations, such as background colors or scene context, rather than learning the actual temporal dynamics of an action. This paper investigates how the fundamental mathematical approach to temporal modeling—convolution, self-attention, or recurrence—influences a model's susceptibility to this texture bias and its overall robustness to domain shifts.
Methodology: Isolating Temporal Modeling
To compare these approaches fairly, the authors stripped away common performance-boosting techniques like heavy pre-training, advanced data augmentation, and dropout. They evaluated three core architectures:
- 3D CNNs: These treat time as a spatial dimension, utilizing linear convolutions across the temporal axis.
- TimeSformer: A Video Transformer that relies on non-linear, time-symmetric self-attention.
- ConvLSTM: A recurrent model using 2D convolutions within a time-causal framework.
The study controlled for layer depth, parameter count, and baseline performance to ensure that differences in robustness could be attributed to the temporal modeling mechanism itself. Models were trained on a source domain and evaluated on held-out target domains to calculate a "robustness ratio."
Datasets and Domain Shifts
The researchers utilized two primary datasets to test their hypotheses. The first is a new synthetic dataset called Temporal Shape (TS), where a simple entity (like a dot or an MNIST digit) follows a specific trajectory. By varying the spatial appearance of the entity or adding noise, the authors could test pure temporal modeling without spatial shortcuts.
The second dataset is a modified version of Diving48, a fine-grained action recognition dataset. To isolate shape from texture, the authors created three test domains:
- S1 (Shape 1): Divers are segmented against a blurred background.
- S2 (Shape 2): Divers are represented by bounding boxes over a blurred background.
- T (Texture): The divers are masked out entirely, leaving only the background context.
Key Results and Discussion
The experiments revealed that recurrence naturally mitigates texture bias. In the Diving48 "T" domain, where the action is invisible, the ConvLSTM's performance dropped significantly. This is the desired behavior for a model that has learned to track motion; if the motion is gone, the accuracy should vanish. In contrast, 3D CNNs and TimeSformers maintained higher relative accuracy in the "T" domain, suggesting they were "cheating" by using background textures to infer the action class.
On the Temporal Shape dataset, both ConvLSTM and TimeSformer showed better generalization to new spatial appearances than 3D CNNs. However, the TimeSformer required significantly more training epochs to converge, likely due to its lack of inductive bias compared to the recurrent model. Furthermore, qualitative analysis of misclassifications showed that ConvLSTM's errors shared more true attributes with the ground truth (e.g., matching takeoff type or somersault count), indicating it learned relevant temporal patterns rather than scene-based guesses.
Conclusion and Future Directions
The findings suggest that incorporating recurrence into spatiotemporal models can bridge the gap between artificial and biological vision systems, the latter of which are rich in recurrent connections and highly robust to domain shifts. While the absolute accuracy in this study was lower than state-of-the-art benchmarks due to the controlled experimental design, the insights provide a clear path for future research. Future work should explore whether pre-training on large datasets like Kinetics or ImageNet alters these fundamental biases in Transformers and CNNs.
Summary updated