Video understanding requires neural networks to capture both spatial layout and temporal progression. Historically, these two dimensions were processed using specialized architectural building blocks: recurrent networks (RNNs) for sequential tracking, or convolutional networks (CNNs) for local feature extraction. Over years of iteration, researchers engineered highly effective "plug-and-play" modules to capture motion efficiently—such as shifting channels across time, computing frame-by-frame differences, or distilling long sequences into keyframes.
However, a fundamental challenge has emerged: these temporal modeling mechanisms are rarely architecture-agnostic. Instead, they are deeply intertwined with the structural assumptions of their underlying backbones. As the field transitions from CNNs to Vision Transformers (ViTs) and State-Space Models (SSMs), many foundational temporal modules fail to transfer seamlessly. Understanding why these mechanisms are bound to specific architectures is critical to building scalable, future-proof video models.
The Mismatch of Inductive Biases
To understand why temporal modules struggle to generalize across architectures, we must look at how different backbones represent data.
Convolutional networks operate on the principles of locality and translation invariance. A 2D or 3D convolutional kernel slides across fixed spatial and temporal grids. Because of this rigid structure, early temporal innovations focused on manipulating these grids directly. For instance, in Temporal Interlacing Network, Hao Shao et al. shift feature channels across neighboring time steps to let a 2D CNN capture temporal dynamics. Similarly, Ziyuan Huang et al. introduced temporally-adaptive convolutions in TAda! Temporally-Adaptive Convolutions for Video Understanding, which calibrate kernel weights based on local frame context.
These methods work beautifully in CNNs because they exploit the local, grid-like nature of the feature maps. But what happens when we move to a Vision Transformer?
Transformers discard the local grid. They flatten a video into a collection of spatio-temporal tokens and compute global pairwise relationships via self-attention. As Zhigang Tu notes in Motion-Driven Visual Tempo Learning for Video-Based Action Recognition, transitioning from local spatial operations to global attention mechanisms fundamentally alters how motion dynamics are captured. A hard-coded channel shift or a local kernel calibration loses its meaning when the architecture treats all tokens—regardless of their spatial or temporal distance—with equal structural fluidity. Consequently, modules optimized for convolutional backbones face severe integration challenges when paired with attention-based structures.
Decompositions, Differences, and Dual Pathways
The dependency on convolutional priors extends to how networks decompose space and time. To avoid the massive computational cost of full 3D convolutions, many frameworks split the problem. Chenxu Luo et al. decomposed 3D kernels along the channel dimension into parallel spatial and temporal paths in Grouped Spatial-Temporal Aggregation for Efficient Action Recognition, while Dongliang He et al. stacked frames into "super-images" combined with lightweight temporal blocks in StNet: Local and Global Spatial-Temporal Modeling for Action Recognition. Other architectures explicitly compute motion boundaries; Limin Wang 0002 et al. utilized a temporal difference operator across frames in TDN: Temporal Difference Networks for Efficient Action Recognition, and Yan Li et al. combined short-range motion excitation with long-range residual aggregation in TEA: Temporal Excitation and Aggregation for Action Recognition.
While these modular decompositions drastically reduce overhead in residual networks, their translation to non-convolutional paradigms remains an open question. For example, a temporal difference operator assumes that subtracting adjacent feature maps yields a clean proxy for motion. In a transformer, where features are dynamically mixed across the entire video via attention, simple frame-level differences may no longer isolate motion cleanly.
This architectural lock-in also affects multi-pathway networks. The iconic SlowFast design by Christoph Feichtenhofer et al. in SlowFast Networks for Video Recognition relies on asymmetric temporal sampling across dual convolutional paths, while the Ample and Focal Network by Yitian Zhang et al. in Look More but Care Less in Video Recognition uses a dynamic selection mechanism to route salient frames through a high-capacity branch. Because these routing and sampling strategies were co-designed with convolutional feature extractors, it is unclear if their benefits translate effectively to the architectural priors of Vision Transformers, which can inherently modulate attention across time without physical dual-path splits.
The Two-Way Street of Parameter-Efficient Adaptation
The cross-architecture generalization problem is not merely about adapting old convolutional tricks to new transformer models; it is a two-way street. Recently, researchers have developed parameter-efficient tuning methods designed specifically for Vision Transformers.
Modules like the ST-Adapter by Junting Pan et al. in ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning and the FE-Adapter by Boyan Gao et al. in FE-Adapter: Adapting Image-based Emotion Classifiers to Videos insert lightweight spatio-temporal bottlenecks into pre-trained image transformers to enable video understanding with minimal parameter updates. However, as Yilong Wang et al. observe in TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognition, these tuning mechanisms exhibit a strong dependency on the specific pre-trained transformer architectures they modify. Attempting to integrate these transformer-centric adapters into alternative backbones—such as convolutional networks or modern State-Space Models (SSMs)—remains highly uncertain.
A similar bottleneck occurs in video generation. Techniques like those proposed by Jiazi Bu et al. in BroadWay: Boost Your Text-to-Video Generation Model in a Training-free Way manipulate temporal attention maps to boost text-to-video generation without retraining. While highly effective for Diffusion Transformers (DiTs), Weilun Feng et al. point out in Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers that temporal consistency mechanisms designed for attention maps require significant adaptation to function in non-attention architectures like SSMs.
Generalizing Training Dynamics and Domain Adaptation
Beyond architectural modules, high-level training paradigms—such as self-supervised pretraining, temporal distillation, and domain adaptation—are also constrained by backbone choices.
Consider temporal distillation. In I Have Seen Enough: A Teacher Student Network for Video Classification Using Fewer Frames, Shweta Bhardwaj et al. validated temporal compression on hierarchical recurrent architectures, while Ping Luo designed plug-and-play compression for 3D CNNs in Temporal Sequence Distillation: Towards Few-Frame Action Recognition in Videos. Whether these distillation losses maintain their performance when compressing the global attention maps of transformers is still an active area of investigation. Similarly, the speed-invariance inductive biases enforced by the temporal contrastive learning framework of Ankit Singh et al. in Semi-Supervised Action Recognition With Temporal Contrastive Learning were optimized for CNNs, leaving their interaction with global self-attention mechanisms largely unmapped.
Even domain adaptation strategies suffer from this dependency. Frameworks designed to align video distributions across different domains—such as the adversarial temporal network by Yuecong Xu et al. in Partial Video Domain Adaptation with Partial Adversarial Temporal Attentive Network, the source-free consistency mechanisms of Yuecong Xu et al. in Source-Free Video Domain Adaptation by Learning Temporal Consistency for Action Recognition, or the open-set contrastive framework by Giacomo Zara et al. in Simplifying Open-Set Video Domain Adaptation with Contrastive Learning—are frequently bound to traditional temporal relation networks or 3D CNNs. When these alignment strategies are forced onto transformer backbones, the way feature clusters form and separate changes entirely, often requiring a complete redesign of the alignment loss functions.
Key Foundational Work
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition (2019) — contributed 1 concept elements to this direction.
SlowFast Networks for Video Recognition (2019) — contributed 1 concept elements to this direction.
I Have Seen Enough: A Teacher Student Network for Video Classification Using Fewer Frames (2018) — contributed 1 concept elements to this direction.