Speech Denoising by Accumulating Per-Frequency Modeling Fluctuations

Michael Michelashvili, Lior Wolf

Abstract We present a method for audio denoising that combines processing done in both the time domain and the time-frequency domain. Given a noisy audio clip, the method trains a deep neural network to fit this signal. Since the fitting is only partly successful and is able to better capture the underlying clean signal than the noise, the output of the network helps to disentangle the clean audio from the rest of the signal. This is done by accumulating a fitting score per time-frequency bin and applying the time-frequency domain filtering based on the obtained scores. The method is completely unsupervised and only trains on the specific audio clip that is being denoised. Our experiments demonstrate favorable performance in comparison to the literature methods. Our code and samples are available at github.com/mosheman5/DNP and as supplementary.

Index Terms: Audio denoising; Unsupervised learning

1. Introduction

Speech denoising is a fundamental task in signal processing with broad applications. Modern solutions typically fall into two categories: supervised models trained on large datasets of clean-noisy pairs, and traditional unsupervised methods relying on heuristics like initial silence for noise estimation. While supervised models are powerful, they often struggle with unseen noise types. Traditional unsupervised methods, such as those using MCRA, fail when noise is non-stationary or when silence is unavailable.

This paper proposes Denoising with Network Priors (DNP), a completely unsupervised approach that requires only the single noisy clip to be denoised. DNP leverages the "Deep Prior" effect—the tendency of neural network architectures to fit structured signals like speech before unstructured noise. Unlike image-based priors, audio networks exhibit significant instability during training. DNP exploits these per-frequency modeling fluctuations to distinguish stable speech from stochastic noise.

2. Proposed Method

2.1 Single-Clip Optimization

For each noisy audio clip, a fresh WaveUnet is initialized and trained to map a random noise vector to the noisy signal. Due to the inductive bias of the convolutional architecture, the network naturally captures the harmonic structure of speech more effectively than noise in the early stages of training.

Evolution of the log-frequency power spectrogram over iterations (25, 100, 250, 1000), demonstrating the network's preference for fitting speech structure early in training.

2.2 Accumulating Fluctuations

As training continues, the network's reconstruction of speech remains relatively stable, while its attempt to model noise fluctuates wildly across iterations. By monitoring the Short-Time Fourier Transform (STFT) of the output, the authors accumulate these absolute differences (fluctuations) for each time-frequency bin.

Visualization of the fluctuation mechanism. Top row (a-d): spectrograms at iterations 2000-2003. Bottom row (e-g): absolute differences between consecutive iterations highlighting unstable noise regions. (h): the target clean signal.

2.3 Mask Generation and Filtering

The accumulated fluctuations are used to create a spectral mask. High-fluctuation bins are identified as noise, while low-fluctuation bins are identified as speech. This mask serves as an estimate of the a-priori SNR and is fed into a classical MMSE-LSA estimator to produce the final denoised signal.

3. Experimental Results

3.1 Quantitative Performance

The method was evaluated on the VoiceBank-DEMAND dataset using standard metrics including PESQ, SSNR, and MOS predictors (CSIG, CBAK, COVL). DNP achieved a PESQ of 2.39, outperforming the previous unsupervised state-of-the-art (MMSE-LSA at 2.38) and the audio-adapted DIP (DAP at 1.99). Notably, DNP performed remarkably close to the supervised SEGAN model (2.16) without any prior training on external datasets.

3.2 Robustness to Silence

In a "No-Silence" test where silent intervals were removed, traditional methods like MCRA-based MMSE-LSA saw a significant performance drop to a PESQ of 1.81. In contrast, DNP maintained a high PESQ of 2.56, demonstrating its robustness in scenarios where noise profiles cannot be estimated from silence.

4. Discussion and Conclusion

DNP provides a robust, unsupervised alternative for speech denoising. However, it currently requires significant computational time (5,000 iterations per clip) and can struggle with unvoiced speech sounds (like "s" or "f") that resemble noise. Future work will explore explicit modeling of unvoiced speech and methods to accelerate the per-clip optimization process. Despite these challenges, DNP represents a significant step forward in unsupervised signal separation by utilizing the internal dynamics of neural network training.

Summary updated

Abstract

We present a method for audio denoising that combines processing done in both the time domain and the time-frequency domain. Given a noisy audio clip, the method trains a deep neural network to fit this signal. Since the fitting is only partly successful and is able to better capture the underlying clean signal than the noise, the output of the network helps to disentangle the clean audio from the rest of the signal. This is done by accumulating a fitting score per time-frequency bin and applying the time-frequency domain filtering based on the obtained scores. The method is completely unsupervised and only trains on the specific audio clip that is being denoised. Our experiments demonstrate favorable performance in comparison to the literature methods. Our code and samples are available at github.com/mosheman5/DNP and as supplementary. Index Terms: Audio denoising; Unsupervised learning
BibTeX
@misc{michelashvili2019speech,
  author = {Michael Michelashvili and Lior Wolf},
  title = {{Speech Denoising by Accumulating Per-Frequency Modeling Fluctuations}},
  year = {2019},
  eprint = {1904.07612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.07612},
}
Figure