Phase Transition Adaptation

Claudio Gallicchio, Alessio Micheli, Luca Silvestri

Abstract

Artificial Recurrent Neural Networks are a powerful information processing abstraction, and Reservoir Computing provides an efficient strategy to build robust implementations by projecting external inputs into high dimensional dynamical system trajectories. In this paper, we propose an extension of the original approach, a local unsupervised learning mechanism we call Phase Transition Adaptation, designed to drive the system dynamics towards the ‘edge of stability’. Here, the complex behavior exhibited by the system elicits an enhancement in its overall computational capacity. We show experimentally that our approach consistently achieves its purpose over several datasets.

Introduction

Reservoir Computing (RC) has emerged as a compelling framework for Recurrent Neural Networks (RNNs), effectively bypassing the vanishing gradient problem by utilizing fixed, random recurrent connections. In this paradigm, only the linear output layer—the readout—requires training. However, the performance of these reservoirs is notoriously sensitive to hyperparameters such as spectral radius and input scaling.

A reservoir's dynamics typically fall into two regimes: "ordered," where the system forgets too quickly, or "chaotic," where it becomes noisy and unpredictable. The optimal state for computation lies at the edge of stability (or edge of chaos). Traditionally, identifying this critical point requires exhaustive manual tuning or grid searches. This work introduces Phase Transition Adaptation (PTA), a local, unsupervised rule that enables a reservoir to autonomously converge to this optimal dynamical state.

Methods: Phase Transition Adaptation

PTA is an online, unsupervised algorithm designed to drive a reservoir toward the edge of stability by adjusting internal neuron parameters.

Base Architecture

The proposed method utilizes a Simple Cycle Reservoir (SCR). In this topology, neurons are arranged in a single ring. This specific structure is mathematically advantageous because it ensures all eigenvalues of the system's Jacobian matrix have the same module, simplifying the calculation of stability through Local Lyapunov Exponents (LLEs).

Simple Cycle Reservoir (SCR) architecture used as the base for PTA.

The Adaptation Rule

PTA introduces two trainable parameters for each neuron: a gain ($a$) and a bias ($b$). The algorithm monitors LLEs, which quantify how small perturbations propagate through the system. Using Stochastic Gradient Descent (SGD) with momentum, PTA adjusts the gains and biases to push the LLEs toward zero. An LLE of zero indicates the exact phase transition—the edge of stability—maximizing the system's capacity for information storage and complex processing. Crucially, this process is local and unsupervised, requiring no target labels.

Experimental Results

The efficacy of PTA was evaluated against standard Echo State Networks (ESN) and SCRs across several benchmarks, including Memory Capacity (MC), Nonlinear Memorization (NLM), NARMA-20, and Mackey-Glass (MG) chaotic time-series prediction.

Memory and Task Performance

In Memory Capacity tests, PTA-tuned reservoirs achieved scores of approximately 95 for a 100-neuron reservoir, nearing the theoretical limit of $N$. This demonstrates that the edge of stability is indeed the optimal regime for short-term memory.

On more complex tasks, PTA showed significant improvements. Notably, in the Nonlinear Memorization (NLM) task, PTA reduced the Normalized Mean Squared Error (NMSE) by more than an order of magnitude compared to standard ESN and SCR models.

Performance comparison across NLM, NARMA, and MG tasks showing PTA's superior error reduction.

Efficiency and Convergence

The adaptation process is computationally efficient, with a complexity of $O(NU)$, matching a standard reservoir update. Experimental results indicate that reservoir dynamics typically reach the optimal regime in fewer than six epochs of unsupervised adaptation.

Discussion and Future Work

While PTA effectively drives reservoirs to the edge of stability, it currently relies on the ring topology of the SCR to maintain mathematical tractability. Future research will focus on extending this local adaptation rule to more complex architectures, such as fully connected or deep reservoirs. Additionally, while PTA mitigates sensitivity to initial conditions, some dependence on initial input scaling remains. The local nature of PTA makes it particularly suitable for implementation in physical reservoir systems, such as optical or electronic hardware, where autonomous operation is critical.

Summary updated

Abstract

Artificial Recurrent Neural Networks are a powerful information processing abstraction, and Reservoir Computing provides an efficient strategy to build robust implementations by projecting external inputs into high dimensional dynamical system trajectories. In this paper, we propose an extension of the original approach, a local unsupervised learning mechanism we call Phase Transition Adaptation, designed to drive the system dynamics towards the `edge of stability'. Here, the complex behavior exhibited by the system elicits an enhancement in its overall computational capacity. We show experimentally that our approach consistently achieves its purpose over several datasets.
BibTeX
@inproceedings{gallicchio2021phase,
  author = {Claudio Gallicchio and Alessio Micheli and Luca Silvestri},
  title = {{Phase Transition Adaptation}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=K17EilcudLA},
}
Figure