Abstract
Malfunctioning neurons in the brain sometimes operate synchronously, reportedly causing many neurological diseases, e.g. Parkinson's. Suppression and control of this collective synchronous activity are therefore of great importance for neuroscience, and can only rely on limited engineering trials due to the need to experiment with live human brains. We present the first Reinforcement Learning gym framework that emulates this collective behavior of neurons and allows us to find suppression parameters for the environment of synthetic degenerate models of neurons. We successfully suppress synchrony via RL for three pathological signaling regimes, characterize the framework's stability to noise, and further remove the unwanted oscillations by engaging multiple PPO agents.
Introduction
Neurological disorders such as Parkinson’s disease are frequently associated with "pathological synchrony," where groups of neurons fire in unison, manifesting as tremors and rigidity. Deep Brain Stimulation (DBS) is a common treatment that delivers electrical pulses to disrupt this activity. However, current DBS is primarily open-loop, firing at constant rates regardless of the brain's real-time state. Developing closed-loop systems is hindered by the risks of experimenting on human subjects and the non-linear complexity of neuronal interactions. This work introduces a standardized, synthetic environment—a "gym"—to safely develop and test data-driven control algorithms for DBS.
Methods and Framework
The researchers developed a custom environment compatible with the OpenAI Gym interface to simulate neuronal dynamics and test RL agents.
Neuronal Modeling
The framework implements two foundational computational neuroscience models:
- Bonhoeffer–van der Pol (BvP) oscillator: Used to simulate regular and chaotic firing regimes.
- Hindmarsh–Rose (HR) model: Used to simulate bursting regimes.
The environment generates synthetic time-series signals of mean-field activity from ensembles of $N=1000$ simulated neurons using a Runge-Kutta-based ODE solver.
Reinforcement Learning Architecture
The core of the framework utilizes Proximal Policy Optimization (PPO). The agent's state is defined by the 250 most recent values of the neuronal mean field. The action space consists of the amplitude of the suppression pulse. The reward function incentivizes the reduction of mean field variance (desynchronization) while penalizing high-energy pulses.
For complex chaotic signals, a multi-agent approach was employed: one agent handles initial strong synchrony, while a second agent provides fine-tuned control once the signal is partially suppressed.
Key Insights and Results
The PPO agent demonstrated model-agnostic control, successfully suppressing synchrony without prior knowledge of the underlying differential equations. This versatility was proven across three distinct pathological patterns: Regular, Chaotic, and Bursting.
Performance Metrics
- Suppression Efficiency: The authors achieved a suppression coefficient ($S$) of approximately 33 for strongly synchronized systems, representing a significant reduction in pathological signaling.
- Convergence: In regular regimes, the agent achieved steady-state suppression within roughly 200 time steps.
- Energy Efficiency: By implementing a "skipping pulse" method (intervening every $\kappa=5$ steps), the agent maintained effective control while significantly reducing energy consumption—a critical factor for battery-powered medical implants.
- Robustness: The system remained resilient to state noise (sensing errors), though it showed higher sensitivity to action noise (delivery errors).
Discussion and Future Work
While this framework provides a robust benchmark using standard ODE models, these are simplifications of biological tissue. Future research should integrate more biophysically detailed models, such as Hodgkin-Huxley. Additionally, the sensitivity to action noise highlights the need for high-precision hardware in clinical applications. Translating these algorithms into low-power, embedded medical hardware and improving the interpretability of the RL agents' policies remain key challenges for regulatory approval and clinical deployment.
Summary updated