Abstract
We explore the robustness of recurrent neural networks when the computations within the network are noisy. One of the motivations for looking into this problem is to reduce the high power cost of conventional computing of neural network operations through the use of analog neuromorphic circuits. Traditional GPU/CPU-centered deep learning architectures exhibit bottlenecks in power-restricted applications, such as speech recognition in embedded systems. The use of specialized neuromorphic circuits, where analog signals passed through memory-cell arrays are sensed to accomplish matrix-vector multiplications, promises large power savings and speed gains but brings with it the problems of limited precision of computations and unavoidable analog noise. In this paper we propose a method, called Deep Noise Injection training, to train RNNs to obtain a set of weights/biases that is much more robust against noisy computation during inference. We explore several RNN architectures, such as vanilla RNN and long-short-term memories (LSTM), and show that after convergence of Deep Noise Injection training the set of trained weights/biases has more consistent performance over a wide range of noise powers entering the network during inference. Surprisingly, we find that Deep Noise Injection training improves overall performance of some networks even for numerically accurate inference.
Introduction
Deep learning's high power consumption is a significant barrier to its deployment in edge applications like mobile speech recognition or embedded sensors. Neuromorphic computing offers a solution by performing matrix-vector multiplications using analog circuits. By representing weights as conductances and inputs as voltages, these systems leverage Ohm’s law for high-efficiency computation. However, analog hardware introduces internal computational noise and limited precision. Unlike traditional robustness research focused on noisy inputs, this work addresses noise generated within the network layers. Standard Recurrent Neural Networks (RNNs) are highly sensitive to such internal errors, necessitating new training strategies.
Methodology: Deep Noise Injection
The core proposal is Deep Noise Injection, a technique that introduces random Gaussian noise during the training phase to build resilience. In a standard layer, the output is $f(x) = g(W \cdot x + b)$. The authors modify this by adding a noise term $z \sim \mathcal{N}(0, \sigma^2)$ immediately after the matrix-vector multiplication:
$$f(x) = g(W \cdot x + b + z)$$
This forces the backpropagation algorithm to find weights and biases that maintain signal integrity despite internal "static." The method was evaluated on Vanilla RNNs and Long Short-Term Memory (LSTM) networks.
Datasets and Tasks
The models were tested on the MNIST handwritten digit dataset using two sequential representations:
- Image-based MNIST: 28x28 images processed row-by-row (28 steps).
- Stroke-based MNIST: Digits represented as sequences of 2D pen coordinates, a more challenging task for learning temporal dependencies.
Results and Discussion
The primary goal of Deep Noise Injection is to achieve a "flat" performance curve across varying levels of inference noise. Quantitative results show dramatic improvements in robustness.
As shown in the figure above, conventionally trained LSTMs ($\sigma_{train} = 0$) experience a catastrophic drop in accuracy (from 98.7% to 12.5%) as validation noise increases. In contrast, models trained with $\sigma_{train} = 1.0$ maintain accuracy above 98% across the entire noise spectrum. Similar trends were observed for Vanilla RNNs and the harder stroke-based MNIST task.
Key Insights
- Weight Adaptation: Analysis of weight distributions suggests that noise-injected models develop weights with higher variance. The network effectively increases its internal "signal power" to overcome the injected noise.
- Regularization Bonus: Surprisingly, injecting noise during training often improved performance on perfectly precise digital hardware. This indicates that Deep Noise Injection serves as a powerful regularizer, preventing overfitting.
- Signal Evolution: In LSTMs, internal representations adapt over time, with signals in later cells becoming more spread out to survive cumulative noise across the sequence.
Conclusion and Future Work
Deep Noise Injection training enables RNNs to operate reliably on power-efficient but noisy analog hardware. While highly effective for MNIST-scale tasks, future work should address convergence challenges in very long sequences and evaluate the method against more complex, non-Gaussian noise profiles typical of real-world neuromorphic devices. Scaling these techniques to larger industrial tasks like natural language processing remains a key next step.
Summary updated