The core mechanism of the Transformer—dense self-attention—suffers from a fundamental scaling problem. Because every token attends to every other token, the computational and memory complexity scales quadratically ($O(N^2)$) with the sequence length. When dealing with long documents, high-resolution images, or extended video sequences, this quadratic bottleneck quickly becomes unsustainable.
To bypass this limitation, early architectures turned to local window attention, restricting each token's focus to its immediate neighbors. While this reduces complexity to a linear scale ($O(N)$), it introduces a severe trade-off: models become short-sighted, losing the ability to capture long-range dependencies and global context.
Dilated attention resolves this conflict. By introducing structured gaps into the attention matrix, dilated attention allows tokens to skip intermediate elements and sample context across vast distances. This simple modification expands the model's effective receptive field exponentially without increasing the number of keys and values processed, maintaining linear efficiency while restoring global awareness.
The Intuition: Exploiting Data Redundancy
To understand why dilated attention works, we must look at the nature of the data we feed into neural networks. Whether processing text, audio, or video, natural signals are highly redundant. Neighboring pixels in an image, consecutive frames in a video, or adjacent waveforms in audio share massive amounts of overlapping information.
If a model attends to every single element in a dense neighborhood, it wastes computational budget recalculating highly correlated relationships. Dilated attention capitalizes on this redundancy by "striding" or "combing" through the sequence.
Instead of attending to a dense local window of size $W$, a dilated attention mechanism samples tokens at a fixed interval or dilation rate $D$. This expands the window's span to $W \times D$. The number of operations remains exactly the same as standard local attention, but the model can now "see" much further down the sequence.
Scaling Temporal Horizons in Audio and Video
The benefits of this expanded view are most pronounced in temporal sequences like audio and video, where long-term context is vital but dense processing is prohibitively expensive.
In speech processing, Chenfeng Miao et al. demonstrate in A compact transformer-based GAN vocoder how a dilated sliding window mechanism allows a lightweight model to capture broad temporal audio context. Similarly, Roshan Sharma et al. apply dilation to speech summarization in End-to-End Speech Summarization Using Restricted Self-Attention, proving that skipping tokens within the attention window significantly improves long-range context awareness without inflating computational costs.
For video understanding, the redundancy across consecutive frames is even more pronounced. Palash Goyal et al. utilize dilated temporal masking in Leveraging Local Temporal Information for Multimodal Scene Classification to sample frames at regular intervals, broadening the temporal receptive field effortlessly.
Taking this further, Hao Zhang et al. introduce "Leap Attention" in Long-term Leap Attention, Short-term Periodic Shift for Video Classification. By grouping video frames into discrete pairs separated by fixed intervals, the authors reduce the quadratic complexity of full temporal attention to a linear scale, demonstrating that dense attention across every single frame is fundamentally unnecessary for video comprehension.
When sequences stretch into tens of minutes, standard architectures break down entirely. To combat this, Emad Bahrami et al. develop a sparse temporal attention mechanism in How Much Temporal Long-Term Context is Needed for Action Segmentation? that partitions long videos and samples a single representative frame from each segment. This sparse sampling strategy enables the model to capture global dependencies across extended durations, slashing training times from weeks to days.
This capacity to scale temporal windows is echoed by Cilin Yan et al. in LTCA: Long-range Temporal Context Attention for Referring Video Object Segmentation, where dilated window attention is combined with random sampling to strike an optimal balance between local precision and global context.
Capturing Global Spatial Context
Beyond 1D temporal sequences, dilated attention translates naturally to 2D and 3D spatial domains, acting as a content-aware, dynamic alternative to dilated convolutions.
In image super-resolution and restoration, capturing both fine-grained local textures and broad structural context is essential. Chengxing Xie et al. show in MAT: Multi-Range Attention Transformer for Efficient Image Super-Resolution that introducing gaps into the attention receptive field allows models to exploit image redundancy and integrate sparse global context efficiently.
For tasks like shadow removal, where a model must understand the relationship between a shadowed region and distant uncorrupted areas, Hengxing Liu et al. employ dilated local attention in Regional Attention for Shadow Removal. This allows shadowed pixels to sample informative features from distant non-shadowed neighbors without increasing the parameter count.
Standard Local Window (Dense)
[X][X][X][X][X][X][X] -> Receptive Field: 7 tokens
Dilated Window (Stride = 2)
[X]._.[X]._.[X]._.[X] -> Receptive Field: 13 tokens
In higher-dimensional spaces, such as 3D medical imaging, the computational savings are even more dramatic. Yixuan Wu et al. introduce D-Former: A U-shaped Dilated Transformer for 3D Medical Image Segmentation, which alternates between local and dilated global attention. By selecting patches at fixed strides across a spatial volume, the model captures long-range 3D spatial dependencies without the catastrophic memory overhead of dense global attention.
Mitigating the Gridding Effect
While dilated attention offers massive efficiency gains, sparse sampling introduces a unique vulnerability: gridding artifacts. Because the attention mechanism strictly skips tokens at fixed intervals, it can create blind spots, completely ignoring the information contained in the skipped elements.
To counteract this, researchers often pair or hybridize dilated attention with local operations to ensure no data is lost. For instance, Abdul Muqeet et al. demonstrate in Multi-attention Based Ultra Lightweight Image Super-Resolution that combining multiple dilated outputs through element-wise addition effectively mitigates gridding artifacts.
By layering or alternating different dilation rates and mixing them with local feature aggregation, modern architectures ensure that the network maintains a continuous, unbroken understanding of the input space while reaping the benefits of a vast receptive field.
Key Foundational Work
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents (2021) — contributed 1 concept elements to this direction.
Multi-attention Based Ultra Lightweight Image Super-Resolution (2020) — contributed 1 concept elements to this direction.
Leveraging Local Temporal Information for Multimodal Scene Classification (2021) — contributed 1 concept elements to this direction.