Simple and Effective Stochastic Neural Networks

Tianyuan Yu, Yongxin Yang, Da Li, Timothy M. Hospedales, Tao Xiang

Abstract

Stochastic Neural Networks (SNNs) are powerful tools for improving model generalization and robustness. However, traditional approaches like Bayesian Neural Networks (BNNs) or Variational Information Bottleneck (VIB) models are often computationally expensive and task-specific. We introduce the Simple and Effective Stochastic Neural Network (SE-SNN), a streamlined framework that models activation uncertainty via a max-entropy regularizer. By adopting a "noisy by default" philosophy, SE-SNN provides a unified solution for network pruning, adversarial defense, label noise robustness, and model calibration. Experimental results on MNIST and CIFAR benchmarks demonstrate that SE-SNN often outperforms more complex Bayesian or Information Bottleneck approaches while maintaining high efficiency.

Introduction

Deep learning models frequently struggle with over-parameterization, sensitivity to adversarial attacks, and poor calibration. While Stochastic Neural Networks (SNNs) offer a principled way to address these issues by incorporating randomness, their adoption is hindered by training complexity and a lack of versatility. Most existing SNN formulations are designed for a single objective, such as pruning or defense, requiring different implementations for different problems.

The SE-SNN framework addresses this by providing a "Swiss Army Knife" for stochastic modeling. It is built on a "noisy by default" philosophy: using a non-informative prior, the network assumes every neuron is uninformative (infinite noise) unless the task specifically requires certainty. This forcing of the network to "earn" its precision leads to naturally sparse and robust representations.

Methodology

The core innovation of SE-SNN lies in its treatment of activations. In a standard deterministic layer, an input produces a fixed output. In an SE-SNN layer, the network predicts both a mean ($\mu$) and a standard deviation ($\sigma$) for each neuron.

Architecture of an SE-SNN layer, where activations are sampled from a learned Gaussian distribution.

Stochastic Layers and Regularization

During the forward pass, activation values are sampled from a Gaussian distribution $N(\mu, \sigma^2)$ using the reparameterization trick, ensuring the model remains end-to-end differentiable. The training objective includes a Max-Entropy Regularizer derived from a non-informative prior. This term encourages the model to maximize the entropy (log-variance) of its activations.

Unified Framework for Multiple Tasks

The versatility of SE-SNN comes from how this entropy term is aggregated:

  • Network Pruning: By aggregating entropy over samples for each neuron, the model identifies and prunes neurons that remain consistently noisy.
  • Adversarial Defense: Inflating entropy across both neurons and samples creates a highly stochastic model that is difficult for gradient-based attacks to penetrate.
  • Label Noise Robustness: Aggregating entropy over neurons for each sample allows the model to automatically down-weight hard-to-classify, noisy samples.
  • Model Calibration: Learned per-sample variance is used to scale the "temperature" of the final softmax, significantly improving confidence estimates.

Experimental Results

The authors evaluated SE-SNN on standard computer vision benchmarks, including MNIST (LeNet-5), CIFAR-10 (VGG-16), and CIFAR-100 (ResNet-110).

Adversarial Robustness

SE-SNN demonstrates superior resilience against adversarial perturbations. It significantly outperforms competitors like Deep VIB and Bayesian Adversarial Learning (BAL) under both Fast Gradient Sign (FGS) and Carlini-Wagner (CW-L2) attacks, particularly as the attack magnitude ($\epsilon$) increases.

Adversarial robustness of SE-SNN compared to baselines under FGS and CW-L2 attacks.

Efficiency and Pruning

On MNIST, SE-SNN achieved the lowest FLOPs (0.061 Mil) and memory footprint (11.08%) among all tested methods while maintaining a 0.9% error rate. For CIFAR-10, it successfully reduced the VGG-16 model size to just 2.57% of its original parameters.

Calibration and Label Noise

By employing "Sample-Wise Temperature Scaling" (SW Temp), SE-SNN reduced the Expected Calibration Error (ECE) by 12-50% compared to standard temperature scaling. Furthermore, the model remained competitive with state-of-the-art label-cleaning methods even without explicit data preprocessing, showcasing its inherent robustness to corrupted labels.

Conclusion and Future Work

SE-SNN offers a simple yet powerful framework for building robust and efficient neural networks. By unifying multiple challenges under a single information-theoretic objective, it simplifies the deployment of stochastic models. Future research will focus on the sensitivity of hyperparameters like the entropy loss weight, exploring non-Gaussian activation distributions, and scaling the approach to larger datasets like ImageNet and complex architectures like Transformers.

Summary updated

Abstract

Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, robustness against adversarial attack and label noise, and model calibration. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense, learning with label noise, and model calibration.
BibTeX
@inproceedings{yu2021simple,
  author = {Tianyuan Yu and Yongxin Yang and Da Li and Timothy M. Hospedales and Tao Xiang},
  title = {{Simple and Effective Stochastic Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16436},
}
Figure