In the standard paradigm of Reservoir Computing, a Recurrent Neural Network (RNN) acts as a fixed, high-dimensional dynamical system. The "reservoir" is typically initialized randomly and left untouched; only the weights of the output layer (the readout) are trained. While this approach is computationally efficient, it assumes that a random initialization will naturally possess the dynamics necessary to solve a specific task.
A growing body of research suggests that this "fixed reservoir" assumption is a bottleneck. By extending optimization beyond the output weights to the internal parameters governing the reservoir's dynamics, networks can adapt to unknown signal ranges, solve complex control problems, and converge faster.
The Importance of Global Scaling
The behavior of an Echo State Network (ESN) is largely dictated by its spectral radius—the largest eigenvalue of the reservoir's weight matrix. This single value determines whether the network's internal states decay quickly, persist as useful memory, or explode into chaos.
In Unsupervised learning of echo state networks: balancing the double pole, Fei Jiang and Hugues Berry demonstrate that for unsupervised tasks like the double pole balancing problem, the optimal spectral radius is not a "one-size-fits-all" constant. By using evolutionary algorithms (specifically CMA-ES) to tune the spectral radius and reservoir topology, the authors achieved state-of-the-art results. This suggests that the reservoir's internal structure is not just a passive feature extractor but a critical component that must be aligned with the task's specific temporal demands.
Similarly, in Supervised and Evolutionary Learning of Echo State Networks, the authors show that optimizing these internal parameters allows ESNs to tackle reinforcement learning tasks that are traditionally difficult for standard gradient-based reservoir methods.
Local Adaptability through Activation Amplitudes
While the spectral radius provides a global knob for dynamics, individual neurons also benefit from local tuning. Standard RNNs use fixed activation functions (like $tanh$), which may not match the dynamic range of an incoming signal.
Su Lee Goh addresses this in Recurrent neural networks with trainable amplitude of activation functions by introducing the AARTRL algorithm. This method allows the network to adapt the amplitude of its activation functions in real-time.
The intuition is simple: if the input signal has unknown or changing dynamics, a fixed activation function might saturate or become too shallow. By making the amplitude a trainable parameter, the network can dynamically rescale its internal responses. Su Lee Goh found that this adaptability leads to significantly lower mean squared error and faster convergence, particularly when dealing with non-linear or "colored" noise signals where the dynamic range is not known a priori.
Differentiable Hyperparameters for Complex Tasks
The shift toward trainable internal parameters has also reached modern large-scale applications. In Echo State Speech Recognition, Harsh Shrivastava et al. propose a modification to ESNs where the scaling factors for both the reservoir and the input matrices are treated as trainable hyperparameters.
By making these scaling factors part of the optimization process, the authors allow the network to fine-tune its sensitivity to inputs and its internal temporal depth during training. This approach retains the core benefit of ESNs—fast training of the readout—while providing a differentiable way to "shape" the reservoir to fit the complexities of speech data.
Why It Works: Tuning the Edge of Chaos
The common thread across these papers is the move from static reservoirs to adaptive dynamics. Whether through evolutionary algorithms or real-time gradient descent, tuning parameters like the spectral radius, activation slope, or input scaling allows the network to find the "edge of chaos." This is the regime where the network has enough memory to process history but enough stability to avoid divergent behavior.
By treating the reservoir as a flexible medium rather than a fixed black box, these methods allow recurrent networks to better match the natural frequencies and scales of the data they are meant to model.
Key Foundational Work
Unsupervised learning of echo state networks: balancing the double pole (2008) — contributed 5 concept elements to this direction.
Supervised and Evolutionary Learning of Echo State Networks (2008) — contributed 5 concept elements to this direction.
Recurrent neural networks with trainable amplitude of activation functions (2003) — contributed 5 concept elements to this direction.