Abstract
Stochastic Neural Networks (SNNs) are powerful tools for improving model generalization and robustness. However, traditional approaches like Bayesian Neural Networks (BNNs) or Variational Information Bottleneck (VIB) models are often computationally expensive and task-specific. We introduce the Simple and Effective Stochastic Neural Network (SE-SNN), a streamlined framework that models activation uncertainty via a max-entropy regularizer. By adopting a "noisy by default" philosophy, SE-SNN provides a unified solution for network pruning, adversarial defense, label noise robustness, and model calibration. Experimental results on MNIST and CIFAR benchmarks demonstrate that SE-SNN often outperforms more complex Bayesian or Information Bottleneck approaches while maintaining high efficiency.
Introduction
Deep learning models frequently struggle with over-parameterization, sensitivity to adversarial attacks, and poor calibration. While Stochastic Neural Networks (SNNs) offer a principled way to address these issues by incorporating randomness, their adoption is hindered by training complexity and a lack of versatility. Most existing SNN formulations are designed for a single objective, such as pruning or defense, requiring different implementations for different problems.
The SE-SNN framework addresses this by providing a "Swiss Army Knife" for stochastic modeling. It is built on a "noisy by default" philosophy: using a non-informative prior, the network assumes every neuron is uninformative (infinite noise) unless the task specifically requires certainty. This forcing of the network to "earn" its precision leads to naturally sparse and robust representations.
Methodology
The core innovation of SE-SNN lies in its treatment of activations. In a standard deterministic layer, an input produces a fixed output. In an SE-SNN layer, the network predicts both a mean ($\mu$) and a standard deviation ($\sigma$) for each neuron.
Stochastic Layers and Regularization
During the forward pass, activation values are sampled from a Gaussian distribution $N(\mu, \sigma^2)$ using the reparameterization trick, ensuring the model remains end-to-end differentiable. The training objective includes a Max-Entropy Regularizer derived from a non-informative prior. This term encourages the model to maximize the entropy (log-variance) of its activations.
Unified Framework for Multiple Tasks
The versatility of SE-SNN comes from how this entropy term is aggregated:
- Network Pruning: By aggregating entropy over samples for each neuron, the model identifies and prunes neurons that remain consistently noisy.
- Adversarial Defense: Inflating entropy across both neurons and samples creates a highly stochastic model that is difficult for gradient-based attacks to penetrate.
- Label Noise Robustness: Aggregating entropy over neurons for each sample allows the model to automatically down-weight hard-to-classify, noisy samples.
- Model Calibration: Learned per-sample variance is used to scale the "temperature" of the final softmax, significantly improving confidence estimates.
Experimental Results
The authors evaluated SE-SNN on standard computer vision benchmarks, including MNIST (LeNet-5), CIFAR-10 (VGG-16), and CIFAR-100 (ResNet-110).
Adversarial Robustness
SE-SNN demonstrates superior resilience against adversarial perturbations. It significantly outperforms competitors like Deep VIB and Bayesian Adversarial Learning (BAL) under both Fast Gradient Sign (FGS) and Carlini-Wagner (CW-L2) attacks, particularly as the attack magnitude ($\epsilon$) increases.
Efficiency and Pruning
On MNIST, SE-SNN achieved the lowest FLOPs (0.061 Mil) and memory footprint (11.08%) among all tested methods while maintaining a 0.9% error rate. For CIFAR-10, it successfully reduced the VGG-16 model size to just 2.57% of its original parameters.
Calibration and Label Noise
By employing "Sample-Wise Temperature Scaling" (SW Temp), SE-SNN reduced the Expected Calibration Error (ECE) by 12-50% compared to standard temperature scaling. Furthermore, the model remained competitive with state-of-the-art label-cleaning methods even without explicit data preprocessing, showcasing its inherent robustness to corrupted labels.
Conclusion and Future Work
SE-SNN offers a simple yet powerful framework for building robust and efficient neural networks. By unifying multiple challenges under a single information-theoretic objective, it simplifies the deployment of stochastic models. Future research will focus on the sensitivity of hyperparameters like the entropy loss weight, exploring non-Gaussian activation distributions, and scaling the approach to larger datasets like ImageNet and complex architectures like Transformers.
Summary updated