For years, generating high-fidelity audio—whether speech, singing, or general sound effects—meant breaking the problem into two distinct pieces. First, an acoustic model would translate an input (such as text or musical scores) into an intermediate frequency-domain representation, most commonly a mel-spectrogram. Second, a separate model called a vocoder would take that spectrogram and reconstruct the time-domain audio waveform.
While this division of labor made training manageable, it introduced a fundamental bottleneck. Mel-spectrograms discard phase information and smooth out fine-grained temporal details. Furthermore, the vocoder is typically trained on perfect, ground-truth spectrograms, but at inference time, it must process noisy, imperfect spectrograms generated by the first stage. This "training-inference mismatch" inevitably leads to compounding errors, spectral blurring, and robotic artifacts.
To overcome these limitations, a major research direction has emerged: direct end-to-end waveform synthesis. By mapping source features directly to raw audio samples in a single, unified network, these architectures eliminate intermediate bottlenecks, preserve phase integrity, and optimize the entire pipeline for the ultimate goal—realistic sound.
The Cost of Intermediate Bottlenecks
To understand why direct synthesis works, we must first look at what is lost in traditional pipelines. Early end-to-end generative frameworks like Tacotron, introduced by Yuxuan Wang et al. in Tacotron: Towards End-to-End Speech Synthesis, successfully unified text analysis and acoustic modeling into a single network. However, they still targeted linear or mel-spectrograms, leaving the final waveform generation to classical phase reconstruction algorithms or separate neural vocoders.
As noted by Rodrigo Mira et al. in End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks, relying on intermediate spectrograms frequently introduces robotic artifacts and limits the natural timing of speech. Because frequency-domain transformations compress acoustic data, subtle but crucial human vocal nuances—such as the rapid, non-linear variations in "vocal fry" or sharp, impulsive transients in sound effects—are smoothed away. When a model attempts to reconstruct the waveform from this blurry representation, the lost details cannot easily be recovered.
Eliminating the Training-Inference Mismatch
The second major flaw of two-stage pipelines is error propagation. When an acoustic model and a vocoder are trained independently, they operate on different data distributions. The vocoder learns to map clean, natural spectrograms to waveforms. But during inference, the acoustic model produces synthetic spectrograms containing slight imperfections and over-smoothing.
Direct waveform synthesis solves this by training a unified model on consistent data distributions. For instance, in VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis, Yongmao Zhang et al. utilize a unified latent representation instead of intermediate mel-spectrograms. This prevents the quality degradation caused by the mismatch between separate acoustic and vocoder stages, yielding more stable and natural-sounding singing voices. Similarly, Dan Su et al. demonstrate in DurIAN-E 2: Duration Informed Attention Network with Adaptive Variational Autoencoder and Adversarial Learning for Expressive Text-to-Speech Synthesis that an end-to-end waveform generation approach using an integrated BigVGAN decoder directly mitigates the acoustic feature-vocoder mismatch problem.
By optimizing the entire network end-to-end, the latent space is tuned specifically for final waveform reconstruction, ensuring that any internal errors are accounted for and minimized globally rather than compounding locally.
Paradigms of Direct Waveform Generation
How do models actually map high-level inputs directly to millions of raw audio samples per second? Several generative paradigms have been adapted to operate directly in the time domain:
- Autoregressive Waveform Modeling: The foundational breakthrough in direct synthesis was WaveNet, proposed by Aäron van den Oord et al. in WaveNet: A Generative Model for Raw Audio. By utilizing dilated causal convolutions, WaveNet models raw audio waveforms one sample at a time. Bypassing intermediate parametric representations allows the network to learn complex, non-linear patterns directly from the signal, eliminating the muffled quality associated with traditional models. To handle long-form generation efficiently, Roi Benita et al. introduced DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation, which generates raw waveforms directly in overlapping frames, capturing fine timing details with constant memory usage.
- Diffusion and Score-Based Models: Instead of generating samples sequentially, diffusion models iteratively refine raw noise into structured waveforms. In Full-band General Audio Synthesis with Score-based Diffusion, Santiago Pascual et al. show that score-matching with variance-exploding diffusion avoids the information loss of discrete tokenization or spectral representations, preserving complex transients and high-frequency details in full-band audio. For text-to-speech, Yuan Gao et al. developed E3 TTS: Easy End-to-End Diffusion-based Text to Speech, which maps text embeddings directly to raw audio samples via a 1D UNet, completely bypassing intermediate mel-spectrograms or phonemes.
- Generative Adversarial Networks (GANs) and Normalizing Flows: To achieve real-time parallel generation without complex distillation, researchers have integrated adversarial and flow-based objectives directly into waveform synthesis. Sungwon Kim demonstrated in FloWaveNet : A Generative Flow for Raw Audio that direct maximum likelihood estimation on flow-based architectures enables single-stage, end-to-end training for parallel audio synthesis. Furthermore, as shown by Yongqi Wang et al. in FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis, integrating a GAN directly into the synthesis pipeline prevents the over-smoothing of acoustic features, producing sharper spectral representations and significantly improving perceived naturalness.
Preserving Paralinguistic and Structural Nuance
Beyond improving audio fidelity, direct-to-waveform synthesis excels at preserving the underlying structure and paralinguistic nuances of the source signal. When intermediate representations are removed, the network can exploit direct correlations between the conditioning input and the fine-grained physical properties of the audio signal.
This advantage is evident across diverse input modalities:
- Linguistic Features: In Multi-Task WaveRNN With an Integrated Architecture for Cross-Lingual Voice Conversion, Yi Zhou et al. demonstrate that directly mapping linguistic features to raw waveforms removes the intermediate acoustic bottleneck, achieving significantly higher speech quality and better speaker similarity in voice conversion.
- Neural Signals: In Toward Fully-End-to-End Listened Speech Decoding from EEG Signals, the authors show that a direct mapping from neural activity to raw audio waveforms minimizes error accumulation and better preserves prosodic details like rhythm and intonation, proving that explicit intermediate acoustic features are not strictly necessary for effective speech reconstruction from brain activity.
- Textual and Structural Inputs: Frameworks like EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecture, developed by Chenfeng Miao et al., combine monotonic alignment mechanisms with high-fidelity decoders to map text sequences directly to raw waveforms, reducing overall system latency and simplifying the synthesis pipeline.
By treating waveform generation not as a secondary reconstruction task but as the primary target of a single, unified network, direct end-to-end synthesis fundamentally redefines how machines learn to produce sound.