End to End Learning from Raw Audio Waveforms

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2535 papers771 conceptsfrozen 2026-10-04

For decades, the standard recipe for audio machine learning began with a "picture" of sound. Researchers would transform raw pressure waves into 2D spectrograms or Mel-frequency cepstral coefficients (MFCCs), effectively treating audio as a computer vision problem. However, a growing body of research suggests that these handcrafted features act as a bottleneck, discarding critical information like phase and fine-grained temporal dynamics.

The shift toward end-to-end learning from raw waveforms replaces fixed mathematical transforms with learnable neural layers. By processing the 1D signal directly, models can discover task-specific acoustic features that are often invisible to traditional spectral analysis.

The Problem with Handcrafted Spectrograms

Traditional audio processing relies on the Short-Time Fourier Transform (STFT) to move from the time domain to the frequency domain. While intuitive for humans, this process introduces several limitations:

  1. Information Loss: Spectrograms typically discard phase information, keeping only the magnitude. As Rishabh Ranjan and colleagues note in Uncovering the Deceptions, preserving this phase is vital for detecting sophisticated synthetic speech.
  2. Fixed Resolution: The Heisenberg uncertainty principle dictates a trade-off between time and frequency resolution. A fixed window size might be great for identifying a steady tone but terrible for capturing a transient click. Awni Y. Hannun demonstrated in Learning Multiscale Features Directly from Waveforms that multiscale convolutional front-ends can bypass these rigid trade-offs.
  3. Human Bias: MFCCs are designed based on the human ear's perception. However, a model identifying heart murmurs or bird calls might need to "hear" frequencies or patterns that the human auditory system ignores.

Learning the Filter Bank

The core mechanism of raw waveform models is the 1D convolutional front-end. Instead of using a fixed Fourier basis, the first layers of the network act as a learnable filter bank.

In Acoustic Modelling from the Signal Domain Using CNNs, Daniel Povey and Sanjeev Khudanpur showed that these learned filters can match or exceed the performance of MFCCs. These filters adapt to the data; for instance, in speaker verification, Ha-Jin Yu and Jee-weon Jung developed RawNet, which uses strided convolutions to learn frequency responses tailored specifically to identifying individual voices.

Some researchers take this further by using parametric filters. Mirco Ravanelli introduced methods to use sinc-functions as filters, allowing the model to learn specific band-pass characteristics (like pitch and formants) with very few parameters. Similarly, Yuhang He used learnable filter banks in SoundDet to capture the inter-channel time delays necessary for localizing moving sound objects.

Preserving Temporal Nuance

Raw waveform processing is particularly effective when the "meaning" of the sound is buried in its temporal structure.

  • Deepfake Detection: Chenglong Wang and Jiangyan Yi utilized raw waveforms in TO-Rawnet to detect the subtle, high-frequency artifacts left by AI voice generators—details often smoothed away in spectrograms.
  • Speech Enhancement: Models like SEANet and the work of Szu-Wei Fu in End-to-End Waveform Utterance Enhancement process entire utterances. This avoids the "blocky" artifacts caused by processing audio in short, independent frames and allows the model to maintain temporal continuity.
  • Bioacoustics: In Active Bird2Vec, Lukas Rauch argues that bypassing manual spectrogram tuning is essential for discovering unique environmental signatures that vary across species and habitats.

Efficiency and Hardware Acceleration

Beyond accuracy, raw waveform models offer significant engineering advantages. By eliminating the STFT and inverse STFT, the processing pipeline becomes more streamlined.

Yu-Chen Lin and Yu Tsao 0001 demonstrated in IA-NET that raw waveform architectures can be optimized for edge devices. By using integer-adder neural networks, they achieved a 71.9% reduction in model size and faster inference, proving that end-to-end models are not just for high-performance clusters but are viable for real-time applications like hearing aids and mobile sensors.

Conclusion

The transition from handcrafted spectral features to raw waveform processing represents a move toward "letting the data speak for itself." As Dimitri Palaz showed in Learning linearly separable features, deep CNNs can transform raw signals into highly organized, linearly separable representations. By removing the "spectrogram middleman," researchers are building audio systems that are more robust, more efficient, and capable of hearing the nuances we didn't know we were missing.

Papers in this version

Showing 60 of 535 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 433 papers
  2. Version 1 · 2026-10-04 · 535 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_raw_waveform_feature_learning_13b0c51f,
  title = {raw waveform feature learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/raw-waveform-feature-learning/rdr_13b0c51f160a4dd49eea1d5350918804},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_end_to_end_learning_from_raw_audio_wavef_13b0c51f_v1,
  title = {End to End Learning from Raw Audio Waveforms},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-learning-from-raw-audio-waveforms/rdr_13b0c51f160a4dd49eea1d5350918804/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}