A time-resolved theory of information encoding in recurrent neural networks

Rainer Engelken, Sven GoedekeNeurIPS 2022 · OpenReview

The Challenge of Fast Signals in Slow Networks

Biological and artificial recurrent neural networks (RNNs) face a fundamental trade-off: they must respond instantly to a dynamic world, yet their internal components—neurons and synapses—operate on inherently slow timescales. Furthermore, the complex, recurrent nature of these networks can generate internal "chaos," creating fluctuations that act as noise and drown out external signals.

Previous theories suggested that a "balanced state"—where strong excitation is nearly perfectly canceled by strong inhibition—could help stabilize networks. However, a rigorous mathematical theory explaining how this works in rate-based networks (the standard in AI) during constantly changing input signals was missing. Traditional tools, like standard Dynamic Mean-Field Theory (DMFT), assume the network is in a steady state, which falls short when analyzing how a network processes a moving image or a spoken sentence.

A Non-Stationary Approach

To bridge this gap, the authors developed a Non-stationary Dynamic Mean-Field Theory. Unlike previous methods, this mathematical tool does not assume constant network statistics, allowing researchers to track how the network's statistical descriptors evolve in response to time-varying signals.

Coupled with this, they employed information-theoretic analysis—specifically "spectral coherence"—to calculate the mutual information rate between the input stimulus and the network's population activity. This quantifies exactly how much of the original signal survives the noise of the network's internal chaos. To validate their theory, the authors trained RNNs using standard machine learning techniques on synthetic, statistically well-defined signals, such as Ornstein-Uhlenbeck (OU) processes and Additive White Gaussian Noise (AWGN).

Key Insights: Speed, Scaling, and Modularity

The theoretical framework revealed several striking insights about how RNNs process information:

  • The Speedup Factor: The "tightness" of the excitatory-inhibitory balance (represented by a parameter $b$) acts as a direct multiplier for the network's speed. The effective timescale of the population dynamics becomes $\tau/b$. For instance, if $b=100$, the network responds 100 times faster than its individual neurons.
  • Linear Information Scaling: The theory predicts—and simulations confirm—that the mutual information rate increases approximately linearly with the tightness of the balance. Stronger inhibition doesn't just stabilize the network; it fundamentally transforms it into a higher-bandwidth communication channel.
As the balance parameter increases, the population rate tracks fast input fluctuations much more accurately, leading to a linear increase in mutual information.
  • Spontaneous Modularity: When trained to track multiple independent signals, the network doesn't just form one giant balanced pool. Instead, it spontaneously breaks symmetry, forming local inhibitory clusters dedicated to each input channel. This suggests that modularity in biological brains might be a natural byproduct of optimizing for efficient, balanced encoding.
  • Robustness to Chaos: While internal chaos (driven by high weight heterogeneity) typically degrades information at low frequencies, a tight balance effectively cancels these fluctuations, making the network highly robust.

Balance as a Generic Solution

Remarkably, even when the network wasn't explicitly instructed to achieve a "balanced" state, the training process naturally forced it there. To minimize error on fast-changing signals, gradient descent naturally pushed the network weights toward a state of strong, canceling excitation and inhibition.

Throughout the training process, as the network learns to track the signal, the mutual information between the input and the network's activity dramatically increases.

During training on an auto-encoder task, the network's test error dropped sharply precisely as it developed a "negative outlier eigenvalue"—a mathematical signature indicating the emergence of a fast-tracking mode. This demonstrates that feedforward excitatory input paired with local recurrent inhibition is a generic, naturally emergent circuit motif for encoding time-varying information.

Limitations and Future Directions

While the non-stationary DMFT provides a powerful new lens for understanding RNNs, the authors note a few limitations. The theory is technically exact only in the limit of an infinitely large network ($N \to \infty$); finite-size effects in very small networks might deviate from these predictions. Additionally, the study relies on simplified "rate units," whereas real biological neurons possess complex dendritic trees and spiking mechanisms that could introduce new frequency-dependent dynamics.

Future work will need to extend this theoretical framework to high-dimensional, multivariate input signals and explore how tight balance interacts with tasks requiring long-term memory, where a network might actually need to slow down its dynamics rather than speed them up.

Summary updated

Abstract

Information encoding in neural circuits depends on how well time-varying stimuli are encoded by neural populations. Slow neuronal timescales, noise and network chaos can compromise reliable and rapid population response to external stimuli. A dynamic balance of externally incoming currents by strong recurrent inhibition was previously proposed as a mechanism to accurately and robustly encode a time-varying stimulus in balanced networks of binary neurons, but a theory for recurrent rate networks was missing. Here, we develop a non-stationary dynamic mean-field theory that transparently explains how a tight balance of excitatory currents by recurrent inhibition improves information encoding. We demonstrate that the mutual information rate of a time-varying input increases linearly with the tightness of balance, both in the presence of additive noise and with recurrently generated chaotic network fluctuations. We corroborated our findings in numerical experiments and demonstrated that recurrent networks with positive firing rates trained to transmit a time-varying stimulus generically use recurrent inhibition to increase the information rate. We also found that networks trained to transmit multiple independent time-varying signals spontaneously form multiple local inhibitory clusters, one for each input channel. Our findings suggest that feedforward excitatory input and local recurrent inhibition - as observed in many biological circuits - is a generic circuit motif for encoding and transmitting time-varying information in recurrent neural circuits.
BibTeX
@inproceedings{engelken2022time,
  author = {Rainer Engelken and Sven Goedeke},
  title = {{A time-resolved theory of information encoding in recurrent neural networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=r6_zHM2POTd},
}
Figure