For decades, audio processing relied on a "pre-baked" understanding of sound. Before a neural network ever saw a speech signal, the audio was transformed into a spectrogram or Mel-filterbank—a 2D representation that mimics human hearing by discarding phase information and compressing frequencies. While efficient, this hand-engineering creates a "feature bottleneck" where potentially useful information is lost before the model even begins to learn.
A growing body of research is moving toward raw waveform modeling, where neural networks learn their own acoustic front-ends directly from the time-domain signal. By treating the raw pressure waves as input, these models can discover task-specific features that traditional signal processing might miss.
The Shift from Spectrograms to Learnable Front-Ends
The transition to raw waveforms began with the realization that deep learning could match, and eventually exceed, the performance of hand-crafted features. Andrew W. Senior and Oriol Vinyals demonstrated in Learning the speech front-end with raw waveform CLDNNs that a Convolutional, Long Short-Term Memory, Deep Neural Network (CLDNN) could process raw audio just as effectively as log-mel filterbanks.
The intuition here is simple: if a model can learn the optimal way to filter sound for a specific task—whether it's identifying a speaker or detecting a bird call—it shouldn't be forced to use a generic Fourier Transform. However, as Keunwoo Choi points out in A Tutorial on Deep Learning for Music Information Retrieval, this flexibility comes at a cost. Raw audio has a massive sampling rate (often 16,000 to 44,100 samples per second), making the input sequences much longer and more computationally expensive than their compressed spectral counterparts.
Constraining the Search Space with SincNet
One of the most successful ways to bridge the gap between "raw" and "efficient" is to use parametric filters. Instead of letting a 1D convolutional layer learn every single weight from scratch, researchers like Joachim Fainberg have utilized architectures like SincNet.
In Acoustic Model Adaptation from Raw Waveforms with Sincnet, the first layer of the network is restricted to learning only the low and high cutoff frequencies of band-pass filters. This "physics-inspired" constraint allows the model to process raw waveforms with far fewer parameters, making it highly efficient for tasks like speaker adaptation where data might be scarce.
The Efficiency Bottleneck
Despite the performance gains, raw waveform modeling is often described as a "rich lab's game." Shucong Zhang and colleagues argue in On the (In)Efficiency of Acoustic Feature Extractors for Self-Supervised Speech Representation Learning that the heavy CNN-based front-ends used in models like wav2vec 2.0 are the primary bottleneck in training.
To combat this, researchers are exploring three main strategies:
- Distillation: Yujin Wang showed in Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition that the heavy waveform front-end can be distilled into a lightweight, filterbank-based version, achieving a 2x inference speedup.
- Discrete Tokenization: Instead of processing continuous waves, David Harwath and Wei-Cheng Tseng introduced Codec2Vec, which operates on discrete units from neural codecs. This reduces storage by 16x and accelerates training significantly.
- Temporal Sparsity: In ultra-low-power settings, Kwantae Kim developed DeltaKWS, a chip that mimics the human brain by "skipping" redundant computations in the audio stream, drastically reducing power consumption for always-on devices.
Beyond Speech: Music and Forensics
The move to raw waveforms isn't limited to speech. In music source separation, Alexandre Défossez and Leon Bottou developed Demucs, which operates directly in the time domain to avoid the artifacts often introduced by converting back and forth from spectrograms. Similarly, for forensic tasks like Audio copy-move forgery detection, processing the raw signal allows the model to detect subtle phase discontinuities that occur when a clip is tampered with—details that a standard spectrogram would likely smooth over.
Why It Works: Intuition
The reason raw waveform learning works is that it removes the inductive bias of human hearing. Traditional features like Mel-spectrograms assume that what matters to a machine is exactly what matters to a human ear. But for a model trying to identify a specific speaker in a noisy room (Bingsheng He, X-TaSNet) or extract the physical parameters of a synthesizer (Han Han, PNP Sound Matching), the "discarded" information in a spectrogram—like fine-grained phase relationships—might actually be the key to the task. By learning the front-end, the model builds its own "ear" tailored specifically to the data it needs to process.