Audio signals are fundamentally hierarchical and multi-scale. Consider a spoken sentence: at a millisecond level, a model must capture rapid spectral transitions, precise phonetic boundaries, and transient acoustic events like breaths. At a multi-second level, it must track long-range dependencies such as prosody, emotional intent, and speaker identity across an entire utterance.
Traditional deep learning architectures force an awkward compromise when dealing with this multi-scale structure. Convolutional Neural Networks (CNNs) excel at local feature extraction due to their shift-invariance and localized receptive fields, but they struggle to connect distant frames. Conversely, Transformers capture global context effortlessly via self-attention but lack the inherent inductive bias required to efficiently parse fine-grained local acoustic structures without massive amounts of data.
To resolve this tension, modern speech and audio modeling has converged on a shared architectural strategy: explicitly integrating local and global feature extraction mechanisms into unified, hybrid topologies.
The Conformer Paradigm: Sequential Hybridization
The most prominent realization of this strategy is the Conformer (Convolution-augmented Transformer) architecture. Instead of forcing a choice between local and global modeling, the Conformer applies both sequentially within a single processing block.
A standard Transformer block passes tokens through a self-attention layer followed by a feed-forward network (FFN). The Conformer modifies this structure by inserting a depthwise separable convolutional module between the self-attention layer and the FFN.
Input ──> [ Macaron FFN ] ──> [ Self-Attention ] ──> [ Convolution Module ] ──> [ FFN ] ──> Output
This specific ordering allows the self-attention mechanism to focus entirely on global relationships across the sequence, while the subsequent convolutional layer sharpens the model's focus on immediate local acoustic structures. As demonstrated by Chung-Cheng Chiu et al. in Pushing the Limits of Non-Autoregressive Speech Recognition, this hybrid design optimizes the model's ability to process speech by capturing both global context and local acoustic structure simultaneously. Similarly, Minseung Kim et al. show in iDeepMMSE: An improved deep learning approach to MMSE speech and noise power spectrum estimation for speech enhancement that this dual capability is crucial for tracking the continuous, highly correlated nature of speech power spectra.
Alternative Topologies: Parallel Branches and Dual Paths
While sequential interleaving is highly effective, researchers have explored alternative topologies to optimize computational efficiency, inference speed, and feature fusion.
Parallel Branches
Instead of stacking convolution and attention sequentially, the E-Branchformer architecture processes the input through parallel branches simultaneously. In OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer, Yifan Peng et al. demonstrate that a parallel branch encoder captures local features through convolutions and global dependencies through self-attention at the same time. This dual-path approach allows the model to integrate fine-grained acoustic details with long-range context more effectively than standard sequential architectures, while offering significant speedups during inference.
Dual-Path Attention
In tasks like speech separation, where processing long sequences frame-by-frame is computationally prohibitive, researchers have replaced convolutions entirely with multi-scale attention mechanisms. In Attention is All You Need in Speech Separation, Cem Subakan et al. introduced the SepFormer, which decomposes sequence modeling into local and global stages by applying transformers within small segments and across those segments, respectively.
Taking this a step further, Qiquan Zhang et al. developed Ripple attention in Ripple sparse self-attention for monaural speech enhancement. By combining high-resolution local attention with coarse-grained dilated global attention, they proved that a detailed view of immediate neighbors paired with a sparse sampling of distant frames is sufficient for high-quality signal reconstruction, drastically lowering computational complexity.
Universality Across the Audio Landscape
The principle of local-global integration has become a foundational blueprint across diverse audio applications, proving that the structural requirements for modeling speech, music, and general environmental audio are largely interchangeable.
- Speech Synthesis (TTS): In Bilingual and Code-switching TTS Enhanced with Denoising Diffusion Model and GAN, Huai-Zhe Yang et al. utilize a Conformer architecture within a diffusion model's encoder to improve the modeling of complex temporal and spectral relationships. For sample-efficient synthesis, Justin Lovelace et al. design a hybrid architecture in Sample-Efficient Diffusion for Text-To-Speech Synthesis that integrates a 1D U-Net for multi-scale feature downsampling with a central Transformer backbone to capture local acoustic details alongside long-range prosody and rhythm.
- Speaker Diarization: The "who spoke when" problem requires tracking rapid speaker transitions while maintaining long-term speaker identity. David Palzer demonstrates in Improving Neural Diarization through Speaker Attribute Attractors and Local Dependency Modeling that a convolution-augmented transformer architecture successfully captures both long-range context and fine-grained temporal dependencies essential for identifying rapid speaker transitions.
- Ultra-Low-Bitrate Speech Coding: When compressing audio to extreme limits, models often distort speaker identity. In Ultra-Low-Bitrate Speech Coding with Pretrained Transformers, Ali Siahkoohi et al. show that combining global context from Transformers with local feature extraction from convolutional networks preserves speaker identity and phonetic accuracy at bitrates as low as 600 bits per second.
By explicitly balancing the inductive bias of local operations with the expansive reach of global attention, these hybrid architectures mirror the inherent multi-scale nature of sound itself, making them the dominant paradigm in modern audio deep learning.