Audio Denoising with Deep Network Priors

Michael Michelashvili, Lior Wolf

Abstract We present a method for audio denoising that combines processing done in both the time domain and the time-frequency domain. Given a noisy audio clip, the method trains a deep neural network to fit this signal. Since the fitting is only partly successful and is able to better capture the underlying clean signal than the noise, the output of the network helps to disentangle the clean audio from the rest of the signal. This is done by accumulating a fitting score per time-frequency bin and applying the time-frequency domain filtering based on the obtained scores. The method is completely unsupervised and only trains on the specific audio clip that is being denoised. Our experiments demonstrate favorable performance in comparison to the literature methods.

Introduction

Speech denoising is a fundamental task in audio processing, traditionally addressed through either supervised deep learning or unsupervised statistical methods. Supervised models, while highly effective, require extensive paired datasets and often fail to generalize to unseen noise types. Conversely, unsupervised methods like the Log-Spectral Amplitude (LSA) estimator rely on strong assumptions, such as the presence of initial silence or noise stationarity, which frequently do not hold in real-world scenarios.

This work introduces an unsupervised approach that leverages the "Deep Network Prior" (DNP) concept. We demonstrate that the inherent structural bias of a neural network can be used to separate speech from noise within a single audio clip, without any prior training.

Methodology

The proposed method utilizes a WaveUnet architecture trained to map random noise to a single target noisy audio clip. Unlike the Deep Image Prior (DIP) used in computer vision, where the network output itself serves as the denoised result, we find that the network's output for audio remains noisy. However, the stability of the learning process provides a powerful signal for separation.

Training loss dynamics

As shown in the loss curves, the network initially fits the structured speech components more readily than the stochastic noise. In later iterations, the model's output fluctuates significantly in regions corresponding to noise, while remaining relatively stable for speech components.

The Fluctuation Mask

We quantify this instability by calculating the log-normalized absolute difference between the network's output at consecutive iterations in the time-frequency domain. By accumulating these fluctuations over thousands of iterations, we generate a spectral mask.

Visualizing modeling fluctuations

This mask effectively estimates the a-priori Signal-to-Noise Ratio (SNR). We then apply a classical LSA estimator to the original noisy signal using this deep-learning-derived mask to produce the final denoised audio.

Experimental Results

We evaluated the DNP method on the VoiceBank-DEMAND dataset, which contains diverse speakers and noise environments. Our method significantly outperformed traditional unsupervised baselines (e.g., MCRA, MMSE-LSA) across standard metrics like PESQ and Segmental SNR (SSNR).

Notably, the method proved highly robust to the absence of initial silence—a common failure point for statistical methods. In tests where silence was trimmed, DNP maintained an SSNR of 13.71, whereas traditional methods dropped as low as 5.59. Furthermore, our unsupervised approach achieved results competitive with supervised models like SEGAN.

Conclusion

By treating modeling instability as a feature rather than a bug, we have developed a robust, unsupervised audio denoising framework. While the method currently faces challenges with unvoiced speech (which can resemble noise fluctuations) and high per-clip computational costs, it represents a significant step toward flexible, data-independent speech enhancement. Future work will focus on explicitly modeling unvoiced phonemes to further improve perceptual quality.

Summary updated

Abstract

We present a method for audio denoising that combines processing done in both the time domain and the time-frequency domain. Given a noisy audio clip, the method trains a deep neural network to fit this signal. Since the fitting is only partly successful and is able to better capture the underlying clean signal than the noise, the output of the network helps to disentangle the clean audio from the rest of the signal. This is done by accumulating a fitting score per time-frequency bin and applying the time-frequency domain filtering based on the obtained scores. The method is completely unsupervised and only trains on the specific audio clip that is being denoised. Our experiments demonstrate favorable performance in comparison to the literature methods. Our code and samples are available at github.com/mosheman5/DNP and as supplementary. Index Terms: Audio denoising; Unsupervised learning
BibTeX
@misc{michelashvili2019audio,
  author = {Michael Michelashvili and Lior Wolf},
  title = {{Audio Denoising with Deep Network Priors}},
  year = {2019},
  eprint = {1904.07612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.07612},
}
Figure