Linear Attention for Long-Form Audio Processing

This direction is split into other directions: local windowed self-attention. Its last version stays readable and citable here.

Version 1 of 141 papers52 conceptsfrozen 2026-10-04

The Transformer architecture has revolutionized sequence modeling, but its core mechanism—self-attention—suffers from a "quadratic bottleneck." Because every element in a sequence attends to every other element, the computational cost grows with the square of the sequence length ($O(T^2)$).

In the world of audio, this bottleneck is particularly punishing. While a sentence might contain 50 words, just one second of high-fidelity audio contains 16,000 to 44,100 samples. Processing minutes of audio or high-resolution spectrograms with standard Transformers quickly exhausts GPU memory. To solve this, researchers have developed a suite of "linear attention" techniques that reduce complexity to $O(T)$, enabling real-time streaming and the analysis of long-form recordings.

Restricting the View: Local and Sliding Windows

The most direct way to achieve linear complexity is to acknowledge that in audio, the most important information is often nearby. Instead of looking at the entire recording, each frame only looks at a small "window" of its neighbors.

In the TS3-Codec, Naoyuki Kanda and Jinyu Li use sliding-window attention to create a purely transformer-based audio codec. By restricting attention to a fixed number of preceding frames, the model maintains a constant memory footprint regardless of the audio's total length. This approach is also vital for real-time speech enhancement; Koen Oostermeijer and Jun Du demonstrated that local causal attention allows Transformers to process audio streams on consumer hardware with minimal latency.

Other researchers have made these windows "adaptive." Xuankai Chang and Shinji Watanabe introduced Adaptive Span Self-Attention, where each attention head learns its own optimal context window. This allows the model to use narrow windows for local acoustic details and wider windows for linguistic context, all while keeping the overall complexity linear.

Capturing the Big Picture: Dilated and Hierarchical Attention

While local windows are efficient, they struggle to capture "global" dependencies, such as the repeating structure of a song or the identity of a speaker across a long meeting. To fix this without returning to quadratic costs, researchers use Dilated Self-Attention (DSA) and hierarchical structures.

Dilated attention works like a zoom lens: it looks at distant frames by skipping over the ones in between. In the Beat Transformer, Jingwei Zhao and Gus Xia use DSA to track musical beats across entire tracks. Similarly, Chong Luo and Wenjun Zeng developed the Multi-Scale Group Transformer (MSGT), which combines group-based attention with a hierarchical fusion strategy to process the extremely long sequences required for high-quality speech separation.

Another approach is Quadtree Attention, used by Dongchao Yang and Yuexian Zou. This mechanism selectively focuses high-resolution attention on "important" regions of a spectrogram while using coarse-grained attention for the rest, mimicking how the human ear prioritizes specific sounds in a noisy environment.

Beyond Softmax: Linear Kernels and State-Space Models

A more fundamental shift involves changing the math of attention itself. Standard attention uses a "softmax" function that couples all queries and keys, preventing them from being computed independently. Linear attention replaces this with a kernel function that allows the keys and values to be summed first, effectively turning the attention mechanism into a linear recurrence.

In the Small-E synthesis model, Théodor Lemerle and Nicolas Obin use Gated Linear Attention to increase training throughput and handle longer speech sequences. Similarly, the BASED architecture by Simran Arora and Sabri Eyuboglu uses a Taylor expansion to approximate softmax, allowing the model to maintain a constant-sized state during generation—resulting in up to 24x higher throughput.

Taking this a step further, Saurabhchand Bhati and James R. Glass proposed the State-Space Large Audio Language Model (ssLALM). By replacing attention layers entirely with State-Space Models (SSMs), they achieved linear scaling for both audio perception and language generation, enabling the analysis of long-form audio content that would be impossible for traditional Transformers.

Hybrid Context: Sinks and Latent States

Finally, some researchers use hybrid strategies to bridge the gap between local efficiency and global context. Windowed Sink Attention (WSA), introduced by Yongyi Zang, combines a local sliding window with a small set of "global sink tokens." These sinks act as a shared memory that any frame can write to or read from, providing a global summary at a fraction of the cost of full attention.

In the realm of speaker diarization, David Palzer and Matthew Maciejewski utilize latent self-attention. This technique compresses the audio sequence into a smaller set of latent representations before applying attention, allowing the model to capture speaker patterns across long recordings without the quadratic memory explosion.

By moving from $O(T^2)$ to $O(T)$, these diverse mechanisms—windows, dilations, linear kernels, and state-space layers—have transformed the Transformer from a short-clip processor into a tool capable of "listening" to the world in real-time and at scale.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_attention_for_long_form_audio_pro_906b8074,
  title = {Linear Attention for Long-Form Audio Processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-attention-for-long-form-audio-processing/rdr_906b80746896449cb424351ac7e81743},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_attention_for_long_form_audio_pro_906b8074_v1,
  title = {Linear Attention for Long-Form Audio Processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-attention-for-long-form-audio-processing/rdr_906b80746896449cb424351ac7e81743/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}