Reinforcement Learning Framework for Deep Brain Stimulation Study

Dmitrii Krylov, Remi Tachet des Combes, Romain Laroche, Michael Rosenblum, Dmitry V. Dylov

Abstract

Malfunctioning neurons in the brain sometimes operate synchronously, reportedly causing many neurological diseases, e.g. Parkinson’s. Suppression and control of this collective synchronous activity are therefore of great importance for neuroscience, and can only rely on limited engineering trials due to the need to experiment with live human brains. We present the first Reinforcement Learning (RL) gym framework that emulates this collective behavior of neurons and allows us to find suppression parameters for the environment of synthetic degenerate models of neurons. We successfully suppress synchrony via RL for three pathological signaling regimes, characterize the framework’s stability to noise, and further remove the unwanted oscillations by engaging multiple PPO agents.

Introduction

Neurological disorders such as Parkinson’s disease are often linked to groups of neurons firing in a pathologically synchronized manner. This collective "rhythm" leads to debilitating macro-symptoms like tremors and rigidity. Deep Brain Stimulation (DBS)—implanting electrodes to deliver electrical pulses—is a common treatment, but it currently operates in an "open-loop" fashion. Doctors manually adjust settings based on observed symptoms, and the device just keeps firing at a set rate.

The "holy grail" is closed-loop DBS: a device that senses brain activity and adjusts its pulses in real-time. However, developing these algorithms is incredibly difficult because you cannot ethically perform "trial and error" on a live human brain. This research addresses the need for a standardized, synthetic simulation environment where RL agents can safely learn to suppress these harmful oscillations before being considered for clinical use.

The RL Gym Framework

The researchers built a specialized OpenAI Gym environment that simulates the collective behavior of thousands of interacting neurons. Instead of traditional static datasets, the framework uses synthetic environments based on systems of Ordinary Differential Equations (ODEs). These models—specifically the Bonhoeffer–van der Pol (BvP) for regular/chaotic signaling and the Hindmarsh–Rose (HR) for complex bursting patterns—are standard benchmarks in computational neuroscience for mimicking the electrical properties of brain tissue.

The Reinforcement Learning framework for Deep Brain Stimulation, featuring an environment of neuronal models and multiple PPO agents.

The system utilizes Proximal Policy Optimization (PPO), a robust policy gradient algorithm. The agent observes the "mean field" (the average electrical activity of the neuron population) and decides the amplitude of a suppression pulse to deliver. A key insight of this work is that the RL agent successfully suppressed synchrony without any prior knowledge of the underlying physics or the specific ODEs governing the neurons, suggesting it could handle the unknown biological pathways of real human brains.

Multi-Agent Suppression Strategy

A novel aspect of this work is the use of multiple PPO agents to achieve higher precision. A single agent often leaves behind small, stubborn oscillations. By engaging a second agent once the signal is partially suppressed, the system achieves much higher precision. In this dual-agent setup, a primary agent handles high-amplitude "strong" signals to initiate suppression, while a secondary agent takes over to fine-tune and eliminate weak-amplitude residual oscillations.

Phase portrait showing successful suppression in the chaotic regime using a dual-PPO agent strategy.

Key Results and Robustness

The framework successfully desynchronized neurons in three distinct pathological regimes: regular, chaotic, and bursting. The effectiveness was measured using a suppression coefficient ($S$), defined as the ratio of standard deviation before versus after stimulation. For strongly synchronized systems, the researchers achieved a suppression coefficient of $S \approx 33$, indicating a massive reduction in pathological synchrony. In the regular regime, the system converged to a natural equilibrium point ($X_0 \approx -0.27$) within approximately 1,100 time steps.

Beyond raw suppression, the study highlighted two critical factors for medical implants:

  1. Energy Efficiency: By introducing a "skip parameter," the agent demonstrated that it could remain effective while significantly reducing the total energy delivered—a critical factor for battery-powered devices.
  2. Noise Robustness: The framework proved stable even when "white noise" was added to both the agent's observations and its actions, simulating real-world sensor and stimulator conditions.

Limitations and Future Work

While BvP and HR are respected models, they remain mathematical abstractions. Future work aims to incorporate more biophysically detailed models, such as Hodgkin-Huxley, and spatially-structured networks. Additionally, improving the interpretability of the learned pulse patterns is vital for clinical safety. Moving toward clinical integration will require addressing hardware constraints like electrode placement and real-time processing latencies, potentially through a library of pre-trained agents for different disease stages.

Summary updated

Abstract

Malfunctioning neurons in the brain sometimes operate synchronously, reportedly causing many neurological diseases, e.g. Parkinson’s. Suppression and control of this collective synchronous activity are therefore of great importance for neuroscience, and can only rely on limited engineering trials due to the need to experiment with live human brains. We present the first Reinforcement Learning (RL) gym framework that emulates this collective behavior of neurons and allows us to find suppression parameters for the environment of synthetic degenerate models of neurons. We successfully suppress synchrony via RL for three pathological signaling regimes, characterize the framework’s stability to noise, and further remove the unwanted oscillations by engaging multiple PPO agents.
BibTeX
@inproceedings{krylov2020reinforcement,
  author = {Dmitrii Krylov and Remi Tachet des Combes and Romain Laroche and Michael Rosenblum and Dmitry V. Dylov},
  title = {{Reinforcement Learning Framework for Deep Brain Stimulation Study}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=C4YKz7pa5x2},
}
Figure