Scaling Generative Audio Models to High Sampling Rates

71 concepts66 papers1 proposal

Most state-of-the-art generative audio models suffer from a hidden limitation: they operate at relatively low sampling rates. While a sampling rate of 16 kHz or 24 kHz is sufficient for basic speech recognition, it fails to capture the high-frequency textures, air, and harmonic details essential for professional music production, cinematic distribution, and high-fidelity spatial audio.

As Heiga Zen et al. observed when developing LibriTTS, standard datasets optimized for speech recognition impose a quality ceiling on generative models. To bridge the "fidelity gap" and reach professional studio standards (44.1 kHz or 48 kHz), researchers are developing new architectures designed to overcome the unique computational and mathematical hurdles of high-resolution audio scaling.

The Challenge of High-Dimensional Audio

Why can we not simply increase the output resolution of existing generative models? The barrier is twofold: an explosion in dimensionality and the fragility of high-frequency phase consistency.

At a standard studio sampling rate of 48 kHz, a single minute of uncompressed audio contains nearly 3 million individual data points. For autoregressive models, which generate audio sample-by-sample, this density introduces severe long-term coherence challenges. As Karan Goel et al. point out in It's Raw! Audio Generation with State-Space Models, the sheer sequence length makes it computationally prohibitive for traditional recurrent or convolutional networks to maintain structural harmony over extended durations.

Furthermore, moving to higher sampling rates drastically expands the acoustic feature space. In frequency-domain modeling, capturing full-band audio requires predicting complex phase and amplitude relationships across a much larger number of frequency bins. Yang Ai et al. demonstrate in APNet that maintaining phase consistency across these wider bandwidths becomes a major bottleneck, often resulting in metallic artifacts or training instabilities if modeled directly (Eric Liu et al. in DelightfulTTS). As Jun Du notes in QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation, failing to resolve these high-frequency details directly causes a noticeable loss of acoustic clarity.

Core Mechanisms for High-Fidelity Synthesis

To circumvent these limitations, the field has converged on four primary strategies: latent space modeling, direct frequency-domain prediction, neural audio upsampling, and alternative sequence architectures.

1. Latent Space Diffusion and Flow Matching

Instead of generating millions of raw audio samples directly, modern frameworks compress the audio into a continuous, lower-dimensional latent space before applying generative modeling.

Haohe Liu et al. leveraged this in AudioLDM, demonstrating that optimizing a latent diffusion model avoids the computational burden of raw waveforms while preserving the capacity to reconstruct high-frequency details. To further optimize this process for professional music production, K R Prajwal et al. introduced MusicFlow, which uses a cascaded flow matching approach to natively generate high-frequency, multi-channel stereo outputs without the heavy overhead of autoregressive pipelines.

2. Direct Amplitude and Phase Prediction

To accelerate synthesis and bypass the slow upsampling layers typical of traditional neural vocoders, some architectures operate entirely within the frequency domain. By predicting the amplitude and phase spectra directly at the frame level, models can generate high-resolution audio without ever processing raw, high-dimensional sample arrays. The authors of APNet proved that this frequency-domain shortcut maintains high spectral accuracy while running significantly faster than time-domain upsamplers.

3. Neural Audio Upsampling and Bandwidth Extension

Rather than generating high-fidelity audio from scratch, a highly efficient alternative is to generate a low-resolution signal (e.g., 16 kHz) and pass it through a dedicated neural upsampler to reconstruct the missing high-frequency components.

Junhyeok Lee et al. pioneered this with NU-Wave, adapting diffusion probabilistic models specifically for the audio super-resolution task to generate pristine 48 kHz audio from 16 kHz inputs. To make this process more versatile, Ye-Xin Lu et al. developed MultiStage Speech Bandwidth Extension with Flexible Sampling Rate Control, cascading specialized neural blocks to allow a single model to handle arbitrary source and target sampling rates dynamically.

[Low-Res Audio (16kHz)] 
       │
       ▼
┌─────────────────────────────────────────────────────────┐
│ Multi-Stage Bandwidth Extension (e.g., NU-Wave / MS-BWE) │
│ 1. Predict High-Frequency Amplitude Envelope            │
│ 2. Reconstruct Phase Consistency across New Bands       │
└─────────────────────────────────────────────────────────┘
       │
       ▼
[Full-Band Studio Audio (48kHz)]

4. State-Space Models for Long Contexts

When raw waveform generation is unavoidable, researchers replace standard Transformers or RNNs with architectures explicitly designed for long-range dependencies. In It's Raw! Audio Generation with State-Space Models, the authors introduced SASHIMI, a model that utilizes a stable Hurwitz parameterization and a multi-scale hierarchical structure. This allows the network to process the dense sample arrays required for professional standards while maintaining structural coherence over long temporal horizons.

By shifting the computational burden away from raw, high-frequency time-series generation toward compressed latents, structured state-spaces, and modular bandwidth extension, these techniques allow generative audio models to finally match the fidelity demands of modern commercial media.

Key Foundational Work

Analysis of voice-quality features of speech that expresses 'anger', 'joy', and 'sadness' uttered by radio actors and actresses (2008) — contributed 1 concept elements to this direction.
An efficient segment-based speech compression technique for hand-held TTS systems (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_fidelity_audio_synthesis_6dd4de93,
  title = {high-fidelity audio synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-fidelity-audio-synthesis/rdr_6dd4de9302f74fe887a413db278a91a8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-18},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_fidelity_audio_synthesis_6dd4de93_v1,
  title = {high-fidelity audio synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-fidelity-audio-synthesis/rdr_6dd4de9302f74fe887a413db278a91a8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-18},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake