Abstract We present a method for audio denoising that combines processing done in both the time domain and the time-frequency domain. Given a noisy audio clip, the method trains a deep neural network to fit this signal. Since the fitting is only partly successful and is able to better capture the underlying clean signal than the noise, the output of the network helps to disentangle the clean audio from the rest of the signal. This is done by accumulating a fitting score per time-frequency bin and applying the time-frequency domain filtering based on the obtained scores. The method is completely unsupervised and only trains on the specific audio clip that is being denoised. Our experiments demonstrate favorable performance in comparison to the literature methods. Our code and samples are available at github.com/mosheman5/DNP and as supplementary.
Index Terms: Audio denoising; Unsupervised learning
1. Introduction
Speech denoising is a fundamental task in signal processing with broad applications. Modern solutions typically fall into two categories: supervised models trained on large datasets of clean-noisy pairs, and traditional unsupervised methods relying on heuristics like initial silence for noise estimation. While supervised models are powerful, they often struggle with unseen noise types. Traditional unsupervised methods, such as those using MCRA, fail when noise is non-stationary or when silence is unavailable.
This paper proposes Denoising with Network Priors (DNP), a completely unsupervised approach that requires only the single noisy clip to be denoised. DNP leverages the "Deep Prior" effect—the tendency of neural network architectures to fit structured signals like speech before unstructured noise. Unlike image-based priors, audio networks exhibit significant instability during training. DNP exploits these per-frequency modeling fluctuations to distinguish stable speech from stochastic noise.
2. Proposed Method
2.1 Single-Clip Optimization
For each noisy audio clip, a fresh WaveUnet is initialized and trained to map a random noise vector to the noisy signal. Due to the inductive bias of the convolutional architecture, the network naturally captures the harmonic structure of speech more effectively than noise in the early stages of training.
2.2 Accumulating Fluctuations
As training continues, the network's reconstruction of speech remains relatively stable, while its attempt to model noise fluctuates wildly across iterations. By monitoring the Short-Time Fourier Transform (STFT) of the output, the authors accumulate these absolute differences (fluctuations) for each time-frequency bin.
2.3 Mask Generation and Filtering
The accumulated fluctuations are used to create a spectral mask. High-fluctuation bins are identified as noise, while low-fluctuation bins are identified as speech. This mask serves as an estimate of the a-priori SNR and is fed into a classical MMSE-LSA estimator to produce the final denoised signal.
3. Experimental Results
3.1 Quantitative Performance
The method was evaluated on the VoiceBank-DEMAND dataset using standard metrics including PESQ, SSNR, and MOS predictors (CSIG, CBAK, COVL). DNP achieved a PESQ of 2.39, outperforming the previous unsupervised state-of-the-art (MMSE-LSA at 2.38) and the audio-adapted DIP (DAP at 1.99). Notably, DNP performed remarkably close to the supervised SEGAN model (2.16) without any prior training on external datasets.
3.2 Robustness to Silence
In a "No-Silence" test where silent intervals were removed, traditional methods like MCRA-based MMSE-LSA saw a significant performance drop to a PESQ of 1.81. In contrast, DNP maintained a high PESQ of 2.56, demonstrating its robustness in scenarios where noise profiles cannot be estimated from silence.
4. Discussion and Conclusion
DNP provides a robust, unsupervised alternative for speech denoising. However, it currently requires significant computational time (5,000 iterations per clip) and can struggle with unvoiced speech sounds (like "s" or "f") that resemble noise. Future work will explore explicit modeling of unvoiced speech and methods to accelerate the per-clip optimization process. Despite these challenges, DNP represents a significant step forward in unsupervised signal separation by utilizing the internal dynamics of neural network training.
Summary updated