Standard Transformers are often bound by a "context window" defined during training. When a model trained on short sequences encounters a significantly longer input at inference time, its performance typically collapses. This failure stems largely from positional embeddings: the model literally loses its sense of time once it steps outside the coordinates it has seen before.
To overcome this, researchers have developed techniques for temporal length extrapolation. The goal is to design position embeddings that allow a model to maintain coherence and semantic understanding across sequences that are orders of magnitude longer than their training data.
The Problem with Absolute Coordinates
In many early Transformer architectures, position is encoded as a fixed vector added to the input. If a model only learns these vectors for positions 1 through 512, it has no representation for position 513. Even with relative position schemes, the model's attention mechanism often fails to generalize to distances it hasn't practiced.
As Fan et al. observe in the context of 3D facial animation, absolute encodings cause autoregressive models to lose stability once the sequence length exceeds the training window. To solve this, the authors propose periodic positional encodings, which inject temporal information using a repeating cycle or modulo operation. By treating time as a recurring signal rather than a linear progression, the model can process indefinitely long sequences without encountering "unseen" coordinates.
Interpolation and Scaling
A common strategy for extending the context window of existing models is interpolation. Instead of extrapolating to new, larger indices, researchers "stretch" the existing positional grid to fit the longer sequence.
Aydemir et al. and Aydemir et al. demonstrate this in point tracking, where they interpolate temporal positional embeddings at inference time. This allows a model trained on short video clips to maintain tracking coherence over much longer durations. Similarly, Yang et al. apply interpolation to Rotary Position Embeddings (RoPE) to handle streaming video that exceeds the original training context. By scaling the frequencies of the rotary embeddings, as also explored by Yuan et al. for looping video generation, the model maintains global structural consistency across extended contexts.
Combining Periodicity and Trends
In domains like healthcare, time-series data contains both repeating cycles (like heartbeats) and long-term trends (like disease progression). Xu et al. and Song et al. address this by combining rotation matrices with exponential decay.
In this hybrid approach:
- Rotation matrices (similar to RoPE) capture periodic oscillations.
- Exponential decay captures the "momentum" of long-term trends.
This dual mechanism allows the model to generalize to forecasting horizons significantly longer than those seen during training, effectively decoupling the training sequence length from the operational window.
Maintaining Semantic Consistency
A subtle but critical challenge in long-context modeling is that attention scores can become unstable over vast distances. Li et al. theoretically demonstrate that traditional frequency-based embeddings (like RoPE) cause attention scores to oscillate and eventually flip signs over long sequences. This "sign-flipping" prevents the model from matching related tokens if they are too far apart.
To fix this, the authors introduce HoPE (Hybrid of Position Embedding), which zeros out the temporal frequencies in the rotary embedding. By removing the oscillatory component for the temporal dimension, the model maintains semantic consistency across infinite distances, ensuring that a query can always "find" a relevant key regardless of the time elapsed between them.
Practical Applications in Video and Beyond
These extrapolatable mechanisms have enabled a new class of "online" and "streaming" models:
- Video Understanding: Wang et al. show that incorporating RoPE into temporal grounding modules allows a model trained on just 4 frames to extrapolate to 64 frames without losing performance.
- Long-form Generation: Gao et al. use cyclic shifts to ensure relative temporal positions remain interpretable even as a video extends indefinitely. Liu et al. use dynamic RoPE adjustments to maintain stable distances between a "rolling window" of active generation and fixed anchor frames, preventing the visual drift common in long-duration AI videos.
- Extreme Contexts: You et al. utilize periodic replication of embeddings to generate commentary for full 45-minute soccer matches in a single pass, far exceeding the limits of standard fixed-position models.
By moving from static, absolute positions to flexible, periodic, or interpolated representations, these works allow Transformers to treat time not as a finite resource, but as a continuous and extensible dimension.
Key Foundational Work
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2022) — contributed 2 concept elements to this direction.
TimelyGPT: Recurrent Convolutional Transformer for Long Time-series Representation (2023) — contributed 2 concept elements to this direction.