Standard convolutional neural networks (CNNs) process video or sequential data using rigid, fixed temporal windows. Whether utilizing 3D convolutions over raw video or 1D convolutions over sequence features, traditional architectures rely on two core assumptions: temporal translation invariance (the same filter weights apply regardless of when an event occurs) and a fixed grid geometry (frames or elements are sampled at uniform intervals).
However, real-world sequences rarely conform to these constraints. A human action like "throwing a ball" can take two seconds or half a second depending on the context. When a rigid convolutional kernel is slid across a timeline, it struggles to accommodate these variations in speed, duration, and cadence. To capture diverse temporal dynamics efficiently, recent research focuses on making temporal convolutions adaptive—allowing networks to dynamically alter their receptive fields and filter weights on the fly.
Elastic Time via Temporal Deformability
One powerful way to break the rigidity of standard convolutions is to make the temporal sampling grid elastic. Instead of forcing a convolutional kernel to sample strictly from fixed, contiguous time steps (such as $t-1, t, t+1$), temporal deformable convolutions allow the network to learn dynamic temporal offsets.
As introduced by Peng Lei et al. in Temporal Deformable Residual Networks for Action Segmentation in Videos, these learned offsets allow a network to adaptively sample frames from a sequence rather than relying on fixed intervals. If an action slows down, the offsets expand to capture a wider temporal window; if it accelerates, they contract. This temporal elasticity allows the system to effectively model human behaviors occurring at different speeds without being constrained by fixed look-back periods.
A similar mechanism is utilized by Jingwen Chen et al. in Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning, where adaptive temporal sampling ensures that motion-based features are accurately represented regardless of the activity's tempo. For structured data like 3D human skeletons, Junwu Weng et al. demonstrate in Deformable Pose Traversal Convolution for 3D Action and Gesture Recognition that a recurrent sub-network can predict kernel offsets based on the current pose. This context-aware sampling ensures that the model's receptive field prioritizes the most relevant joints at different stages of a temporal sequence.
Breaking Translation Invariance with Content-Adaptive Weights
Altering where a kernel looks addresses temporal scale, but optimizing what the kernel looks for requires relaxing temporal translation invariance. Instead of sharing a static set of weights across an entire sequence, content-adaptive architectures calibrate or generate convolutional weights dynamically based on the input context.
In TAda! Temporally-Adaptive Convolutions for Video Understanding, Ziyuan Huang et al. propose adaptively calibrating convolution weights for each frame based on its local and global temporal context. This shift from shared weights to adaptive calibration enables standard 2D architectures to effectively process temporal sequences and handle extreme dynamics like rapid camera shake or motion blur. Similarly, the Temporal Adaptive Module (TAM) developed by Zhaoyang Liu et al. in TAM: Temporal Adaptive Module for Video Recognition dynamically adjusts weights based on the specific features of an input video sequence, allowing the model to tailor its sensitivity to unique motion patterns.
Rather than just calibrating existing weights, some frameworks generate entire convolutional kernels on the fly. For instance, Wenhao Wu et al. show in DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning that passing a global summary of an input sequence through a multi-layer perceptron allows the network to adaptively weight and fuse information based on global context. In video salient object detection, Jie Liu et al. design a framework in Dynamic Context-Sensitive Filtering Network for Video Salient Object Detection that generates custom convolution kernels based on the calculated affinity between regions in consecutive frames, replacing static filters with adaptive operations that respond directly to shifting object dynamics.
Conditioned and Multi-Scale Modulation
The conditioning signal that shapes these dynamic filters does not have to come exclusively from the visual features of the sequence. It can also be driven by external modalities or multi-scale structural cues.
In Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos, Yitian Yuan et al. demonstrate that temporal convolutional layers can be dynamically adjusted using scaling and shifting parameters derived from a natural language query. This linguistic context modulates how video features are aggregated and correlated across different time steps, effectively "steering" the network's temporal focus toward the specific event described in the text.
Furthermore, advanced architectures often combine weight adaptation with receptive field adaptation to handle highly complex scenes. In DyFADet: Dynamic Feature Aggregation for Temporal Action Detection, Le Yang et al. introduce a module that adapts both kernel weights and receptive fields simultaneously based on video content. This dual adaptation provides a highly flexible representation that allows temporal models to better handle diverse action scales and varied action lengths across complex scenes.
By transforming temporal convolutions from rigid, static stencils into elastic, content-conditioned filters, these approaches allow deep networks to navigate the fluid and unpredictable nature of time in sequential data.
Key Foundational Work
Structure Preserving Video Prediction (2018) — contributed 2 concept elements to this direction.
Temporal Deformable Residual Networks for Action Segmentation in Videos (2018) — contributed 2 concept elements to this direction.
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition (2015) — contributed 1 concept elements to this direction.