Denoising Noisy Neural Networks: A Bayesian Approach With Compensation

Yulin Shao, Soung Chang Liew, Deniz Gündüz

TL;DR

This paper introduces a novel Bayesian denoising framework ($MMSE_{pb}$) to recover the weights of "Noisy Neural Networks" (NoisyNNs) corrupted by environmental or system noise. By prioritizing the reconstruction of large-magnitude weights—which are critical for model performance—the authors achieve significant accuracy gains (up to 13.4 dB improvement over standard methods) without requiring model retraining.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are increasingly deployed in "noisy" environments. Weights can become corrupted during wireless transmission (e.g., in Federated Edge Learning), when stored on noisy analog hardware, or through aggressive quantization and truncation.

The standard approach to handling these "NoisyNNs" has been Maximum Likelihood (ML) estimation, which essentially treats the noisy observations as the true weights. However, ML estimation is suboptimal because it treats every weight error as equally important. In reality, neural network performance is highly sensitive to large-magnitude weights, while the vast majority of weights are small and less impactful. The authors aim to bridge this gap by creating a denoiser that maximizes inference accuracy rather than just minimizing raw weight error.

Methods

The authors propose a Compensated Minimum Mean Squared Error ($MMSE_{pb}$) denoiser. Their approach moves beyond standard estimation through three key components:

  1. Bayesian Framework: They treat DNN weights as random variables following a Gaussian prior distribution (parameterized by the sample mean and variance of the weights).
  2. Population Compensator ($\lambda$): This is the core innovation. It modifies the standard MSE loss function to "weigh" errors in larger parameters more heavily. Since large weights are rare but vital for accuracy, this compensator ensures they are reconstructed with higher fidelity.
  3. Bias Compensator ($\beta$): An additional temperature parameter that corrects for potential shifts in the weight distribution caused by the noise process.

For complex, modern architectures where analytical solutions are impossible, the authors use a grid search on a tiny validation dataset (as few as 10–80 examples) to find the optimal "temperature" parameters ($\lambda, \beta$). They also introduce layerwise denoising, allowing different layers to have unique compensation parameters for better flexibility.

Datasets

The researchers evaluated their approach on two primary tasks:

  • Image Classification: Using the CIFAR-10 dataset (60,000 images across 10 classes) with ResNet34, ResNet18, and ShuffleNet V2 architectures.
  • Sentiment Analysis (NLP): Using the SST-2 dataset (part of the GLUE benchmark) with a BERT model.

Key Insights and Ideas

  • Accuracy $\neq$ Fidelity: The "aha moment" of the paper is that minimizing the Mean Squared Error (MSE) of the weights does not necessarily maximize the accuracy of the model. A denoiser should care more about the weights that drive the model's decisions.
  • The Power of Priors: Even a simple Gaussian approximation of weight distributions provides enough "prior information" to significantly outperform ML estimation, which assumes no prior knowledge of weight structure.
  • Post-Hoc Utility: Unlike "noise-resilient training" (which requires retraining the model with noise injection), this denoiser can be applied at the receiver or deployment stage. This makes it highly practical for base stations or edge devices that receive pre-trained models over noisy channels.

Key Results

The $MMSE_{pb}$ denoiser consistently outperformed ML estimation across all tests:

  • Noisy Inference (ResNet34 on CIFAR-10): The denoiser outperformed ML estimation by 4.1 dB to achieve a 60% test accuracy.
  • Noisy Training (ResNet18 in a Federated Learning setup): The gains were even more dramatic. To reach 60% and 80% accuracy, the denoiser outperformed ML by 13.4 dB and 8.3 dB, respectively.
  • NLP (BERT on SST-2): Even for complex transformer models, the denoiser provided a 1.1 dB improvement in noise tolerance to maintain a 75% accuracy level.
  • Theoretical Validation: The authors provided rigorous mathematical proofs for quadratic and shallow feedforward networks, showing that their compensated approach is analytically superior to standard ML.

Limitations and Future Work

  • Parameter Tuning: The method relies on finding optimal values for $\lambda$ and $\beta$. While the authors show this can be done with a very small validation set, it still adds a "grid search" step to the deployment pipeline.
  • Layerwise Complexity: While layerwise denoising improves performance, it increases the number of hyperparameters to tune, which could become computationally expensive for extremely deep networks.
  • Noise Assumptions: The current framework is optimized for Additive White Gaussian Noise (AWGN). While many physical noise sources (like thermal noise) fit this model, the performance under non-Gaussian or structured noise (like adversarial interference) remains an open question for future research.

Summary updated

Abstract

Deep neural networks (DNNs) with noisy weights, which we refer to as noisy neural networks (NoisyNNs), arise from the training and inference of DNNs in the presence of noise. NoisyNNs emerge in many new applications, including the wireless transmission of DNNs, the efficient deployment or storage of DNNs in analog devices, and the truncation or quantization of DNN weights. This article studies a fundamental problem of NoisyNNs: how to reconstruct the DNN weights from their noisy manifestations. While prior works relied exclusively on the maximum likelihood (ML) estimation, this article puts forth a denoising approach to reconstruct DNNs with the aim of maximizing the inference accuracy of the reconstructed models. The superiority of our denoiser is rigorously proven in two small-scale problems, wherein we consider a quadratic neural network function and a shallow feedforward neural network, respectively. When applied to advanced learning tasks with modern DNN architectures, our denoiser exhibits significantly better performance than the ML estimator. Consider the average test accuracy of the denoised DNN model versus the weight variance to noise power ratio (WNR) performance. When denoising a noisy ResNet34 model arising from noisy inference, our denoiser outperforms ML estimation by up to 4.1 dB to achieve a test accuracy of 60%. When denoising a noisy ResNet18 model arising from noisy training, our denoiser outperforms ML estimation by 13.4 dB and 8.3 dB to achieve test accuracies of 60% and 80%, respectively.
BibTeX
@article{shao2023denoising,
  author = {Yulin Shao and Soung Chang Liew and Deniz Gündüz},
  title = {{Denoising Noisy Neural Networks: A Bayesian Approach With Compensation}},
  journal = {IEEE Trans. Signal Process.},
  year = {2023},
  doi = {10.1109/tsp.2023.3290327},
}
Figure