Listening to Sounds of Silence for Speech Denoising

Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, Changxi Zheng

Speech denoising—the process of separating a speaker's voice from background interference like traffic, wind, or chatter—is a fundamental yet notoriously difficult challenge in audio processing. Traditional methods, such as the classic "spectral subtraction" technique found in commercial audio software, often require users to manually select a "silent" segment of the recording to sample the background noise. However, these manual methods struggle significantly with non-stationary noise (noise that changes over time) because a single static sample cannot capture dynamic environmental sounds.

While modern deep learning approaches have improved denoising performance, they often treat the neural network as a "black box," ignoring the inherent structures of audio data. In this paper, the authors ask a pivotal question: What natural structures of human speech can we leverage to build better, more interpretable architectures?

Their solution centers on a simple but powerful observation: human speech is naturally intermittent. Between sentences and words, there are frequent, albeit short, pauses. These natural "silent intervals" provide brief windows where only the background noise is present.

Waveforms of clean and noisy speech highlighting natural silent intervals.

By automatically detecting these noise-only windows and "inpainting" the noise profile across the entire signal, the authors introduce a deep learning framework that achieves state-of-the-art results, remarkably outperforming even complex audiovisual models that rely on video data.

Methodology: A Three-Stage Pipeline

The authors propose a pipeline that integrates the intuition of classical signal processing with the predictive power of deep neural networks. The architecture is divided into three distinct stages:

The proposed three-stage deep learning pipeline for speech denoising.
  1. Silent Interval Detection: The first stage utilizes a 2D Convolutional Neural Network (CNN) encoder paired with a bidirectional LSTM to process the audio spectrogram. Its goal is to identify segments where the speaker is silent, outputting a "confidence mask" that indicates which parts of the signal consist purely of background noise.
  2. Noise Estimation (The "Inpainting" Step): Because these silent intervals are sparse and intermittent, the model must estimate what the noise looks like underneath the active speech. Borrowing the concept of "inpainting" from computer vision, the model treats noise estimation as a task of filling in the blanks. It exploits local correlations in the spectrogram to create a continuous, time-varying estimate of the background noise.
  3. Noise Removal: Finally, a third network takes the original noisy spectrogram and the estimated full noise profile to produce a complex ratio mask. This mask is applied to the noisy signal to filter out the interference. An Inverse Short-Time Fourier Transform (ISTFT) is then used to reconstruct the final, clean waveform.

While the model can be trained end-to-end—where silence detection emerges naturally as the network attempts to minimize denoising loss—the authors found that a two-step supervised approach using ground-truth silence labels yielded the best overall performance.

Experimental Setup and Key Insights

To rigorously test the model, the authors utilized the AVSPEECH dataset (featuring 4.5 hours of clean English speech) and mixed it with diverse noise sources from the DEMAND (environmental/transportation noise) and Google’s AudioSet datasets. They evaluated the model across various Signal-to-Noise Ratios (SNRs), ranging from a relatively clear 10dB down to an extremely noisy -10dB.

Several fascinating insights emerged from the research:

  • Silence is Information: The core breakthrough is a paradigm shift: treating silence not as a lack of data, but as the most reliable source of information about the surrounding noise environment.
  • Universal Speech Structure: Although the model was trained exclusively on English speech, it generalized remarkably well to Chinese, Japanese, and Korean. This suggests that the temporal distribution of pauses is a fundamental, cross-lingual characteristic of human speech.
  • Emergent Behavior: Even without explicit supervision (i.e., without being told where the pauses are), the network naturally learns to identify silent intervals simply by optimizing for the cleanest possible audio output.

Results and Performance

The model was evaluated against several heavyweights in the field, including SEGAN, DFL, and Adobe Audition’s commercial noise reduction tool.

Under the PESQ metric (Perceptual Evaluation of Speech Quality), the proposed model consistently outperformed all audio-only baselines. On the DEMAND dataset, it achieved a PESQ of 2.795, significantly higher than SEGAN's 2.227. Furthermore, the model maintained high performance even at -10dB SNR, a severe noise level where many competing algorithms fail to produce intelligible speech.

Perhaps most impressively, the model outperformed VSE, a state-of-the-art audiovisual model that uses both audio and video (lip movement) of the speaker. This demonstrates that intelligently exploiting the natural structure of an audio signal can be more effective than introducing entirely new data modalities.

Limitations and Future Directions

While highly effective, the method does have limitations. It fundamentally relies on the presence of pauses. In extremely crowded environments—such as the classic "cocktail party" scenario with multiple overlapping speakers—finding a truly "silent" interval where only background noise exists becomes exceedingly difficult. Additionally, while it generalizes well, the authors note the model might still carry slight biases toward the specific pausing patterns of the languages it was trained on.

Looking forward, the authors suggest extending this "silent interval" logic beyond human speech. The same principles of detecting baseline noise during intermittent signal pauses could be applied to denoising other non-audio time-series signals, such as gravitational wave data or medical telemetry.

Summary updated

Abstract

We introduce a deep learning model for speech denoising, a long-standing challenge in audio analysis arising in numerous applications. Our approach is based on a key observation about human speech: there is often a short pause between each sentence or word. In a recorded speech signal, those pauses introduce a series of time periods during which only noise is present. We leverage these incidental silent intervals to learn a model for automatic speech denoising given only mono-channel audio. Detected silent intervals over time expose not just pure noise but its time-varying features, allowing the model to learn noise dynamics and suppress it from the speech signal. Experiments on multiple datasets confirm the pivotal role of silent interval detection for speech denoising, and our method outperforms several state-of-the-art denoising methods, including those that accept only audio input (like ours) and those that denoise based on audiovisual input (and hence require more information). We also show that our method enjoys excellent generalization properties, such as denoising spoken languages not seen during training.
BibTeX
@inproceedings{xu2020listening,
  author = {Ruilin Xu and Rundi Wu and Yuko Ishiwaka and Carl Vondrick and Changxi Zheng},
  title = {{Listening to Sounds of Silence for Speech Denoising}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=7_7eWsD6Bpj},
}
Figure