Novel Reinforcement Learning Algorithm for Suppressing Synchronization in Closed Loop Deep Brain Stimulators

Harsh Agarwal, Heena Rathore

TL;DR

This paper introduces a novel reinforcement learning (RL) algorithm designed to suppress pathological neural synchronization in Parkinson's disease patients using closed-loop Deep Brain Stimulation (DBS). By combining a temporal stimulus representation with a Twin-Delayed Deep Deterministic (TD3) policy gradient approach, the algorithm achieves more effective and efficient suppression of abnormal brain oscillations compared to existing RL methods, while also reducing power consumption.

What Problem Does This Paper Address?

Parkinson's disease is characterized by abnormal and increased firing patterns in the brain, leading to pathological oscillations and synchronization. Deep Brain Stimulation (DBS) is a treatment that can help regulate these oscillations. While machine learning, particularly RL, has been explored for optimizing DBS, current methods often require significant training data and computational power, which are limitations for resource-constrained DBS devices. Furthermore, existing RL approaches for DBS can suffer from issues like high variance in target values, leading to less effective suppression and higher convergence points. This paper aims to address these challenges by developing a more efficient and effective RL framework for closed-loop DBS that can suppress pathological synchronization with reduced power consumption.

Methods

The core of the proposed solution is a novel RL algorithm built upon the Twin-Delayed Deep Deterministic (TD3) policy gradient algorithm. This algorithm is an off-policy RL method known for its stability and effectiveness in continuous control tasks. The authors introduce two key enhancements:

  1. Temporal Representation of Stimuli: This component aims to improve learning by mimicking Pavlovian conditioning. It breaks down the neuronal stimulus into smaller parts with time differences, creating a more sustained neuronal activity representation. This is hypothesized to enhance learning, especially in scenarios where rewards might be unpredictable or delayed. The equation for this is given as: $X(t) = \sum_{n=1}^{3} 0.33 * (Z_n(t))$ where $Z_n(t)$ represents time-differed components of the signal $X(t)$.

  2. Target Policy Smoothing Regularization: This technique, inspired by TD3, addresses the issue of high variance in target values often seen in deterministic policy methods. It works by adding a small amount of noise to the target action and averaging over mini-batches. This regularization helps to ensure that similar actions receive similar Q-values, leading to more stable learning and preventing overestimation. The equation for this is: $y = R + \gamma Q_{\theta'}(S', \text{clip}(\pi_{\phi'}(S') + \epsilon, -c, c))$ where $\epsilon$ is the added noise.

Additionally, the paper highlights the use of a Twin Critic Network approach, which is a standard component of TD3, to reduce bias in Q-value estimation. The reward function is designed to penalize large actions and encourage convergence to the mean field of previous states, defined as: $R(t) = -\frac{100}{3} (\frac{1}{10} (\sum_{i=1}^{3} (X(t) - [\bar{X}_{state}]_{t})^2) + \frac{1}{10})$

Datasets

The paper does not use traditional datasets in the sense of pre-collected data. Instead, it utilizes simulated environments based on established neuronal models to train and evaluate its RL agent. Specifically, it employs:

  • Bonhoeffer-van der Pol (BVP) oscillators: Used to simulate "regular" and "chaotic" signaling regimes.
  • Hindmarsh-Rose (HR) neuronal model: Used to simulate "bursting" signaling regimes.

These models are implemented within a gym environment (referencing [28]), which allows for the simulation of neurons governed by their specific differential equations. This approach allows for controlled experimentation with different pathological oscillation patterns.

Key Insights and Ideas

  • Bridging the Gap in RL for DBS: The paper effectively demonstrates how advanced RL techniques like TD3 can be adapted to address the specific challenges of closed-loop DBS, particularly the need for efficient synchronization suppression with limited computational resources.
  • The Power of Temporal Representation: The novel temporal stimulus representation appears crucial for improving the learning process. By creating a more nuanced representation of the stimulus, the agent can learn more effectively, especially in dynamic and potentially unpredictable neural environments.
  • Balancing Suppression and Power Consumption: While achieving high suppression coefficients, the proposed method also shows a significant reduction in energy supplied to the ensemble compared to other RL algorithms, which is a critical factor for implantable devices.
  • Robustness to Different Oscillation Types: The algorithm's ability to effectively suppress synchronization across regular, chaotic, and bursting regimes highlights its potential for broad applicability in treating various forms of pathological neural activity.

Key Results

The proposed RL algorithm demonstrated superior performance in suppressing neural synchronization across different regimes:

  • High Suppression Coefficients: The algorithm achieved high suppression coefficients, indicating effective reduction of oscillations. For example, a coefficient of $2.490 \times 10^{15}$ was achieved for the regular regime.
  • Faster Convergence: The proposed method showed better suppression capability with fewer time stamps compared to baseline RL algorithms like A2C, ACKTR, and PPO.
  • Reduced Energy Consumption: In the bursting regime, the proposed RL algorithm consumed 49.47% less energy compared to other RL algorithms.
  • Stable Rewards: Evaluation rewards for the proposed RL algorithm in the bursting regime were consistently high and stable, contrasting with the oscillating rewards observed for other algorithms.
  • Mean Point of Convergence: The algorithm achieved a stable mean point of convergence around M ~ -1.005, indicating effective stabilization of neural activity.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Elevated Energy Consumption (Relative): While showing reduced energy consumption compared to other RL algorithms, the paper notes that the proposed method still has elevated energy consumption compared to simpler approaches.
  • Variable Reward Schedules: The current work uses a fixed reward schedule. Future work will explore implementing and studying the effects of variable reward schedules (e.g., variable ratios, fixed schedules, variable schedules).
  • "Brain-on-Chip" Model Testing: The authors plan to test the algorithm on a "brain-on-chip" model to qualitatively study its performance on actual hardware.
  • Reducing Power Consumption: A key future goal is to reduce the power consumption of the RL algorithm without compromising its suppression capabilities, given that RL models are inherently complex and power-intensive.

Summary updated

Abstract

Parkinson's disease is marked by altered and increased firing characteristics of pathological oscillations in the brain. In other words, it causes abnormal synchronous oscillations and suppression during neurological processing. In order to examine and regulate the synchronization and pathological oscillations in motor circuits, deep brain stimulators (DBS) are used. Although machine learning methods have been applied for the investigation of oscillations in DBS, these models require large amounts of training data and computational power, both of which pose challenges to resource-constrained DBS. This research proposes a novel reinforcement learning (RL) framework for suppressing the synchronization in neuronal activity during episodes of neurological disorders with less power consumption. The proposed RL algorithm comprises an ensemble of a temporal representation of stimuli and a twin-delayed deep deterministic (TD3) policy gradient algorithm. We quantify the stability of the proposed framework to noise and reduced synchrony using RL for three pathological signaling regimes: regular, chaotic, and bursting, and further eliminate the undesirable oscillations. Furthermore, performance metrics such as evaluation rewards, energy supplied to the ensemble, and the mean point of convergence were used and compared to other RL algorithms, namely the Advantage Actor Critic (A2C), the Actor Critic with Kronecker-Featured Trust Region (ACKTR), and the Proximal Policy Optimization (PPO).
BibTeX
@inproceedings{agarwal2023novel,
  author = {Harsh Agarwal and Heena Rathore},
  title = {{Novel Reinforcement Learning Algorithm for Suppressing Synchronization in Closed Loop Deep Brain Stimulators}},
  booktitle = {NER},
  year = {2023},
  url = {https://openreview.net/forum?id=u4KvnRKFuo},
}
Figure