Parameter Efficient Speech and Audio Processing

236 concepts218 papers1 proposal

Deep learning has dramatically advanced speech enhancement, source separation, and spoken language understanding. However, deploying these models on resource-constrained edge devices—such as hearing aids, IoT sensors, and mobile processors—introduces strict computational and memory limitations. While the prevailing trend in deep learning favors scaling up parameter counts, a distinct paradigm focuses on architectural efficiency.

The core intuition behind this direction is that audio signals possess rich structural, physical, and mathematical properties. By tailoring neural networks to exploit these domain-specific priors, factorizing high-dimensional feature spaces, and collapsing redundant multi-stage pipelines, researchers can substitute raw parameter scaling with targeted architectural design.


Grounding Architectures in Acoustic and Physical Priors

Standard deep neural networks are often forced to learn the basic physics of sound from scratch, consuming significant parameter capacity simply to model periodic structures and acoustic relationships. Embedding domain-specific acoustic priors directly into the network topology allows models to remain remarkably lean.

Human speech is inherently characterized by harmonic patterns generated by the vocal cords and filtered by the vocal tract. In FFC-SE: Fast Fourier Convolution for Speech Enhancement, Ivan Shchekotov et al. introduce Fast Fourier Convolutions (FFCs) to speech modeling. Because FFCs possess an instant global receptive field in the frequency domain, they capture periodic harmonic structures across the entire spectrum without requiring the extreme depth or width of traditional convolutional layers.

Similarly, Nurali Alip et al. demonstrate in A Three-Stage Beamforming with Harmonic Guidance for Multi-Channel Speech Enhancement that explicitly guiding a network to recognize the periodic, comb-like structure of human vocal cords allows multi-channel beamforming systems to isolate speech in severe noise environments with only marginal increases in parameter size.

This physical grounding can also be achieved by mimicking classical signal processing frameworks. In Efficient Speech Enhancement with Neural Homomorphic Synthesis, Wenbin Jiang et al. leverage the traditional source-filter model of speech production. By using homomorphic filtering to separate an audio signal into its excitation and vocal tract components, the authors design a neural network that operates on each component independently, achieving performance on par with complex-valued networks while cutting the parameter budget in half.

Traditional CNN:  [Deep/Wide Convolutions] ---> Must learn harmonics from data
Efficient Priors: [Acoustic/Harmonic Prior] ---> Instant global context via fewer parameters

When dealing with multi-channel or multi-microphone arrays, parameter scaling typically compounds with each added channel. To circumvent this, Jun Qi et al. propose Tensor-to-Vector Regression for Multi-channel Speech Enhancement based on Tensor-Train Network. Instead of flattening multi-channel inputs into a massive vector—which triggers a parameter explosion in subsequent linear layers—the authors treat the input as a high-dimensional tensor and apply a Tensor-Train decomposition. This structural factorization compresses the network by up to 81% while preserving spatial and spectral correlations.


Factorizing Attention and Feature Correlations

Transformers and Conformers have become dominant in audio processing due to their ability to model long-range dependencies. However, computing global self-attention across every time-frequency bin creates a massive computational bottleneck. Parameter-efficient architectures overcome this by factorizing attention mechanisms along distinct acoustic dimensions.

Instead of computing a single, massive attention matrix over the entire time-frequency spectrogram, efficient models split the workload. For instance, Jian Luo et al. introduce Tiny-Sepformer: A Tiny Time-Domain Transformer Network For Speech Separation, which slashes model size by nearly 90% compared to baseline transformers. It achieves this by factorizing the separation task into smaller, isolated structural blocks that alternate between modeling short-term local interactions and long-term global dependencies.

To achieve real-time execution on consumer hardware, Koen Oostermeijer et al. propose a Lightweight Causal Transformer with Local Self-Attention for Real-Time Speech Enhancement. By restricting the self-attention mechanism to a localized, causal window, the model eliminates the need to look ahead into future frames or back into the distant past, allowing a compact 6-million-parameter model to outperform much larger recurrent baselines.

Global Attention:    [ Every Time-Frequency Bin attends to Every Other Bin ] -> O(N^2) Parameters
Factorized Attention: [ Local Time Window Attention ] x [ Isolated Band/Channel Attention ] -> O(N) Parameters

This factorization principle applies equally to the internal channels of a neural network. In Improving Speech Enhancement by Integrating Inter-Channel and Band Features with Dual-branch Conformer, Jizhen Li et al. argue that treating network channels as independent entities wastes representational capacity. By introducing specialized modules that explicitly model the correlations between channels alongside spectral band features, the authors demonstrate that architectural awareness of feature interactions is far more critical to audio quality than sheer model scale.

At the extreme edge, this multi-path factorization enables models to shrink to sub-megabyte sizes. Haoyin Yan et al. developed LiSenNet: Lightweight Sub-band and Dual-Path Modeling for Real-Time Speech Enhancement, an ultra-lightweight architecture containing only 37,000 parameters. By prioritizing low-frequency resolution (where human speech energy is concentrated) and utilizing dual-path sub-band modeling, the network achieves state-of-the-art denoising on hearing aids and IoT sensors.


Collapsing Multi-Stage Pipelines

Traditional spoken language systems rely on cascaded pipelines: an acoustic front-end cleans the audio, an automatic speech recognition (ASR) engine transcribes it, and a natural language understanding (NLU) module infers intent. Each discrete stage introduces its own parameter overhead and risks propagating errors down the chain. Parameter-efficient speech processing frequently relies on collapsing these cascades into unified, end-to-end models.

In Towards end-to-end spoken language understanding, Dmitriy Serdyuk et al. demonstrate that mapping raw audio features directly to semantic intent bypasses the need for intermediate textual representations. This direct optimization allows their end-to-end model to be roughly 40 times smaller than a standard multi-stage pipeline while maintaining real-time processing speeds.

A similar phenomenon is observed in speech translation. Alexandre Bérard et al. show in End-to-End Automatic Speech Translation of Audiobooks that a single model translating source audio directly to target text requires less than half the parameters of a cascaded system combining separate recognition and translation models.

Cascaded Pipeline:  [Audio] -> [Speech Enhancement] -> [ASR Engine] -> [NLU Engine] -> [Intent]
                                    (Millions of parameters per independent stage)

End-to-End Model:   [Audio] ---------------------------------------------------------> [Intent]
                                    (Shared, compressed representation space)

When multi-task functionality is unavoidable, parameter efficiency can be preserved by appending ultra-lightweight modules to frozen, pre-trained architectures rather than training entirely new models. In Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator, Lingwei Meng et al. address the problem of identifying who spoke when during overlapping conversations. Instead of deploying a massive standalone diarization model, the authors attach a tiny "sidecar" branch requiring only 768 additional parameters to an existing speech separation network. This works because the spatial and speaker-identity masks are already latent within the separation network's features; the sidecar simply extracts them without duplicating the underlying computational architecture.


Distributing Computation via Iterative Refinement

A final core mechanism for parameter efficiency is trading time for space. Instead of resolving a complex audio transformation in a single, parameter-heavy forward pass, compact models can distribute the computational load across time by reusing a small set of weights iteratively.

This mechanism is highly effective in generative and source separation tasks. In A diffusion-inspired training strategy for singing voice extraction in the waveform domain, Genís Plaja-Roglans et al. tackle the problem of isolating vocals from a full musical mix. Instead of using a massive single-pass network, they train a highly compact model (~750,000 parameters) that iteratively refines the audio waveform over multiple steps, transforming a deterministic mixture into clean vocals. By recycling the same lightweight parameters across successive refinement steps, they achieve a perceptual quality that matches single-pass architectures orders of magnitude larger.

For text-to-music generation, K R Prajwal et al. apply a similar philosophy in MusicFlow: Cascaded Flow Matching for Text Guided Music Generation. By replacing heavy autoregressive or diffusion-based architectures with a cascaded flow-matching paradigm, the authors achieve state-of-the-art audio synthesis while keeping the model 5x smaller and 5x faster than contemporary baselines.

Ultimately, these diverse strategies—whether grounding networks in the physics of speech, factorizing multi-dimensional feature spaces, collapsing processing pipelines, or recycling parameters through iterative refinement—converge on a singular insight: high-performance speech and audio processing does not require massive parameter scaling, provided the architecture respects the underlying structure of the acoustic signal.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_efficient_speech_enhancement_f0e2dce2,
  title = {parameter-efficient speech enhancement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-efficient-speech-enhancement/rdr_f0e2dce279954356a0aef682ce670da3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameter_efficient_speech_enhancement_f0e2dce2_v1,
  title = {parameter-efficient speech enhancement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-efficient-speech-enhancement/rdr_f0e2dce279954356a0aef682ce670da3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake