Denoising Noisy Neural Networks: A Bayesian Approach with Compensation

Yulin Shao, Soung Chang Liew, Deniz Gündüz

Abstract

Deep Neural Networks (DNNs) are increasingly deployed in edge environments where weights are susceptible to corruption from wireless transmission noise, analog hardware storage, or aggressive quantization. Traditional recovery methods like Maximum Likelihood (ML) estimation treat all weight errors equally, which is suboptimal given that DNN performance is disproportionately sensitive to large-magnitude weights. This paper introduces a novel Bayesian denoising framework, the Compensated Minimum Mean Squared Error ($MMSE_{pb}$) denoiser. By incorporating a population compensator ($\lambda$) and a bias compensator ($\beta$), our approach prioritizes the reconstruction of critical high-magnitude weights. Evaluations on CIFAR-10 and SST-2 datasets show that $MMSE_{pb}$ achieves up to 13.4 dB improvement in noise tolerance over standard ML estimation without requiring expensive model retraining.

Introduction

The deployment of DNNs at the network edge, particularly in Federated Learning (FL) scenarios, introduces significant challenges regarding model integrity. As illustrated in Figure 1, model updates transmitted over noisy channels (e.g., AWGN) arrive at the base station corrupted. Such "NoisyNNs" exhibit degraded inference accuracy if the noise is not properly mitigated.

System model of noisy weight transmission in Federated Edge Learning.

While standard signal processing techniques often rely on Maximum Likelihood (ML) estimation, this approach is "accuracy-agnostic." It treats the noisy observation as the ground truth, failing to leverage the inherent structure of neural network weights. In contrast, we propose a Bayesian framework that treats weights as random variables, allowing for more sophisticated recovery strategies that prioritize model utility over raw fidelity.

Methodology: Compensated Bayesian Denoising

The core of our proposal is the Compensated Minimum Mean Squared Error ($MMSE_{pb}$) denoiser. Unlike standard MMSE, which minimizes the average squared error across all weights, $MMSE_{pb}$ introduces parameters to "weigh" the importance of different weight magnitudes.

Key Components

  1. Bayesian Prior: We model DNN weights using a Gaussian prior distribution, capturing the statistical characteristics of the pre-trained model.
  2. Population Compensator ($\lambda$): This parameter modifies the loss function to ensure that large-magnitude weights—which are rare but essential for decision-making—are reconstructed with higher precision.
  3. Bias Compensator ($\beta$): A temperature-like parameter that corrects for distribution shifts induced by the noise process.

As shown in Figure 2, the choice of the compensation parameter $\lambda$ defines the estimator's behavior. While $\lambda=0$ corresponds to standard MMSE (zero compensation) and a specific higher value corresponds to ML (exact compensation), our framework allows for "over-compensation" or "under-compensation" to maximize final inference accuracy.

The spectrum of the population compensation parameter $\lambda$ and its relation to standard estimators.

For modern deep architectures, we employ a grid search on a small validation set (10–80 samples) to determine the optimal $(\lambda, \beta)$ values. This process can be further refined through layerwise denoising, where compensation parameters are tuned independently for each layer to account for varying sensitivity.

Experimental Results

We evaluated the $MMSE_{pb}$ denoiser across diverse tasks and architectures, including ResNet and ShuffleNet for image classification (CIFAR-10) and BERT for sentiment analysis (SST-2).

  • Inference Accuracy: In a ResNet34 model on CIFAR-10, $MMSE_{pb}$ outperformed ML estimation by 4.1 dB to maintain a 60% accuracy threshold.
  • Federated Learning: In noisy training scenarios with ResNet18, the gains were more pronounced, reaching up to 13.4 dB improvement over ML.
  • Natural Language Processing: For BERT on SST-2, the denoiser provided a 1.1 dB improvement in noise tolerance, demonstrating its applicability to complex transformer architectures.

Conclusion

The $MMSE_{pb}$ framework represents a significant shift from traditional weight recovery to utility-aware denoising. By recognizing that accuracy $\neq$ fidelity, we provide a practical, post-hoc solution for deploying robust DNNs in noisy environments. Future work will explore the framework's performance under non-Gaussian noise sources and structured interference.

Summary updated

BibTeX
@article{shao2023denoising,
  author = {Yulin Shao and Soung Chang Liew and Deniz Gündüz},
  title = {{Denoising Noisy Neural Networks: A Bayesian Approach with Compensation}},
  journal = {IEEE Trans. Signal Process.},
  year = {2023},
  doi = {10.1109/tsp.2023.3290327},
}
Figure