Inherent Weight Normalization in Stochastic Neural Networks

Georgios Detorakis, Sourav Dutta, Abhishek Khanna, Matthew Jerry, Suman Datta, Emre Neftci

Multiplicative stochasticity, such as Dropout, is widely known to improve the robustness and generalizability of deep neural networks. However, researchers have recently demonstrated that "always-on" multiplicative stochasticity, when combined with simple threshold neurons, provides a sufficient and highly efficient substrate for deep learning. These models, termed Neural Sampling Machines (NSMs), turn noise from a nuisance into a powerful computational feature, naturally performing weight normalization and accelerating convergence.

The Normalization Problem

Training deep neural networks typically requires explicit normalization techniques like Batch Normalization (BN) to prevent internal covariate shift—a phenomenon where the distribution of inputs to a layer changes constantly as previous layers update. While effective, BN has significant drawbacks: it requires processing data in batches, making it ill-suited for real-time online learning, and it is computationally expensive to implement in specialized low-power hardware.

Simultaneously, in the realm of neuromorphic engineering and "In-Memory Computing" (using devices like memristors or RRAM), physical hardware is inherently noisy. Engineers traditionally expend significant effort to mitigate this noise. NSMs flip this paradigm by asking: what if we embraced the noise to solve the normalization problem "for free"?

Neural Sampling Machines (NSMs)

Unlike standard artificial neural networks that rely on smooth activation functions (such as ReLU or Sigmoid), NSMs utilize simple binary threshold units. The core innovation lies in the application of multiplicative noise to the weights or activations.

When a neuron receives many inputs (a large "fan-in"), the Central Limit Theorem dictates the behavior of the sum. Mathematically, the probability of an NSM neuron firing becomes a function of the L2 norm of its weights. This activation probability naturally mirrors Weight Normalization, a well-known technique that decouples the magnitude of a weight vector from its direction.

This architecture yields two major operational benefits:

  1. Multiplication-Free Forward Pass: Inference relies almost entirely on simple additions and comparisons, making it highly efficient for hardware implementation.
  2. Consistent State: Unlike Dropout, which is typically disabled during inference, the noise in NSMs is "always-on." The network behaves identically during both learning and inference phases, a critical requirement for true continual and online learning.

Inherent Normalization and Stability

Because normalization in an NSM is a mathematical consequence of multiplicative noise rather than an explicit algorithmic step, the network exhibits a "self-normalizing" property. This stabilizes weight distributions during learning and effectively mitigates internal covariate shift.

Activation stability over training iterations comparing NSM and StNN.

As shown in the comparison above, the activations in an NSM remain remarkably stable over training iterations. In contrast, standard Stochastic Neural Networks (StNNs) suffer from severe activation drift. This inherent stability allows NSMs to converge much faster than their standard stochastic counterparts.

Furthermore, NSMs exhibit fascinating scale invariance. Because the normalization term in the denominator cancels out weight magnitudes, the network is highly robust to weight fluctuations. In experiments where weights were scaled down by 90% during inference, the NSM maintained its accuracy, whereas standard artificial neural networks failed completely.

Empirical Performance

The researchers evaluated NSMs across a variety of benchmarks to prove their versatility, including standard vision tasks (MNIST, CIFAR-10/100) and neuromorphic, event-based datasets (N-MNIST, DVS Gestures). The latter are particularly important for demonstrating performance on the streaming data typical of low-power edge devices.

NSMs consistently matched or outperformed deterministic networks with the same architecture:

  • MNIST: Achieved a 0.775% error rate, outperforming a standard deterministic CNN (0.880%).
  • DVS Gestures: Reached an 8.56% error rate, significantly better than the deterministic baseline of 9.16%.
Classification error rates on CIFAR-10 and CIFAR-100 datasets.

On more complex datasets like CIFAR-10 and CIFAR-100, variants of the NSM (such as bNSM and gNSM) demonstrated highly competitive error rates, often edging out standard deterministic CNNs (dCNN).

Limitations and Future Directions

While the forward pass of an NSM is multiplication-free and highly efficient, the backward pass (learning phase) remains mathematically complex. Training requires propagating gradients through the neuron's discrete step function by using the derivative of its activation probability function (an erf function). Calculating these L2 norms and derivatives introduces computational overhead.

Although these terms are shared across a neuron's connections to minimize the burden, the authors note that simplifying the learning rule is the next critical hurdle. Approximating these complex backward-pass calculations with simple hardware-friendly primitives could pave the way for a new generation of autonomous, low-power learning machines that natively exploit the physics of their substrates.

Summary updated

Abstract

Multiplicative stochasticity such as Dropout improves the robustness and gener- alizability deep neural networks. Here, we further demonstrate that always-on multiplicative stochasticity combined with simple threshold neurons provide a suf- ficient substrate for deep learning machines. We call such models Neural Sampling Machines (NSM). We find that the probability of activation of the NSM exhibits a self-normalizing property that mirrors Weight Normalization, a previously studied mechanism that fulfills many of the features of Batch Normalization in an online fashion. The normalization of activities during training speeds up convergence by preventing internal covariate shift caused by changes in the distribution of inputs. The always-on stochasticity of the NSM confers the following advantages: the network is identical in the inference and learning phases, making the NSM a suitable substrate for continual learning, it can exploit stochasticity inherent to a physical substrate such as analog non-volatile memories for in memory computing, and it is suitable for Monte Carlo sampling, while requiring almost exclusively addition and comparison operations. We demonstrate NSMs on standard classification benchmarks (MNIST and CIFAR) and event-based classification benchmarks (N-MNIST and DVS Gestures). Our results show that NSMs perform comparably or better than conventional artificial neural networks with the same architecture.
BibTeX
@inproceedings{detorakis2019inherent,
  author = {Georgios Detorakis and Sourav Dutta and Abhishek Khanna and Matthew Jerry and Suman Datta and Emre Neftci},
  title = {{Inherent Weight Normalization in Stochastic Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=aMqb15mmp9c},
}
Figure