Reinforcement Learning for Closed-Loop Propofol Anesthesia: A Human Volunteer Study

AAAI 2010

TL;DR

This paper introduces a reinforcement learning (RL) agent for closed-loop propofol anesthesia delivery, aiming for patient-specific control. The RL agent successfully demonstrated accurate and stable hypnosis control in healthy human volunteers, suggesting its potential for broader clinical application, even in patients with poorly understood drug responses.

What Problem Does This Paper Address?

Achieving precise control of anesthetic depth, specifically propofol-induced hypnosis, is a critical challenge in anesthesia. While closed-loop systems using the bispectral index (BIS) have shown promise, they often rely on population-based models that struggle with individual patient variability. Factors like age, gender, disease, and even surgical events can significantly alter a patient's response to propofol. This variability leads to two opposing risks: intraoperative awareness (if hypnosis is too light) and adverse hemodynamic/respiratory effects (if too deep). The paper addresses the need for a control system that can adapt to these individual differences and maintain a desired level of hypnosis accurately and safely.

Methods

The core of this work is the development and evaluation of a reinforcement learning (RL) agent for closed-loop propofol anesthesia.

  • Agent Architecture: The RL agent takes the patient's bispectral index (BIS) as input, smoothed and processed to derive the error (BIS - BIS target) and its change. Fuzzy membership functions are used to condition these inputs into a six-dimensional feature vector.
  • Action Space: The agent can select an intravenous propofol infusion rate from a discrete set (0.0–6.0 ml/min), with each selection held for five seconds. A key limitation is the asymmetric control – the agent can only increase propofol, not decrease it.
  • Knowledge Representation: A tabular value function approximator is used, where the state space is discretized into bins, creating a large table to represent the agent's control policy.
  • Learning Algorithm: The agent is trained using Watkins' Q-learning algorithm, a model-free, off-policy temporal difference learning method known for its robustness and suitability for tabular function approximators. Eligibility traces are employed to speed up learning.
  • Training: The agent is initially trained on a simulated "in silico" patient. This virtual patient models propofol pharmacokinetics (using Schnider's three-compartment model) and pharmacodynamics (using a nonlinear model trained on empirical data). The simulation involves a large number of training episodes (5x10^7) to achieve clinical readiness.
  • In Silico Evaluation: Before human trials, the agent's control policy is evaluated using a "Patient Variability Model" (PVM) that simulates variations in patient demographics, propofol sensitivity, and intraoperative events. This PVM challenges the agent with scenarios not directly observable by it.
  • Performance Metrics: Standard metrics for closed-loop control are used, including:
  • PE (Performance Error): (BISmeasured – BIStarget) / BIStarget * 100
    • MDPE (Median Performance Error): Control bias.
  • MDAPE (Median Absolute Performance Error): Control accuracy.
  • Wobble: Intra-subject variability.
  • Divergence: Slope of the regression line of MDAPE measurements, indicating stability.
  • Controlled: Percentage of measurements within ±10 BIS of target.
  • RMSE (Root-Mean-Square Error): For maintenance intervals.

Datasets

  • Training Data: The agent is trained using a simulated intraoperative patient. The pharmacokinetics are based on Schnider's three-compartment model, and pharmacodynamics are modeled using a nonlinear model trained on previously obtained data (Doufas et al. 2004).
  • Evaluation Data:
    • In Silico: A population of 1,000 simulated patients with randomized demographic and Patient Variability Module (PVM) parameters.
  • Human Volunteers: Data from fifteen consecutively studied, healthy human volunteers.

Key Insights and Ideas

  • RL for Patient-Specific Anesthesia: The study demonstrates that RL can learn patient-specific control policies, adapting to individual responses to propofol, even when the underlying physiological models are imperfect or unknown to the agent.
  • Robustness to Variability: The RL agent showed good control performance across a range of simulated patient variations and in healthy human volunteers, suggesting its potential for populations with less predictable PK/PD profiles.
  • Asymmetric Control Challenge: The agent's inability to actively decrease propofol concentrations (asymmetric control) is a notable limitation, leading to "soft landings" that might not be ideal for time-sensitive surgical schedules.
  • Potential for Broader Application: The success in healthy volunteers, coupled with the simulation results, points towards the possibility of applying this RL approach to more complex patient populations where traditional models are less effective.

Key Results

  • Clinical Performance: In healthy human volunteers, the RL agent achieved aggregate control metrics that were within or close to desired performance values (e.g., MDPE < ±5.0%, MDAPE < 7.5%, Wobble < 5.0%, Divergence ≤ ±0.1%/hr, Controlled ≥ 80%, RMSE < 5.0 BIS).
  • Specific Metrics:
    • For BIStarget = 40: MDPE = 1.0±5.6%, MDAPE = 7.4±3.5, Wobble = 6.2±2.6.
  • For BIStarget = 60: MDPE = -0.2±1.2%, MDAPE = 2.8±1.2, Wobble = 2.6±1.2.
  • Aggregate: MDPE = 0.4±3.0%, MDAPE = 5.1±1.7, Wobble = 4.5±1.5.
  • Controlled (aggregate): 82.9±9.6%.
  • Patient-Specific Examples: Figure 5 shows examples of the agent compensating for apparent propofol sensitivity (Volunteer A) and tolerance (Volunteer B), demonstrating its adaptive capabilities.
  • Induction Time: The agent's reward structure, which discouraged overshoot, resulted in induction times averaging 12.5 minutes, which may be longer than desired for some surgical procedures.

Limitations and Future Work

  • Limitations:
    • Controlled Environment: The study was conducted in healthy volunteers, not in the full rigor of an intraoperative surgical setting. This limits the direct extrapolation of results to surgical patients.
  • Asymmetric Control: The agent can only increase propofol infusion, not decrease it, leading to potentially slower inductions ("soft landings").
  • PK/PD Model Dependence: While the RL agent aims to overcome reliance on perfect models, its initial training is based on a specific PK/PD model.
  • Future Work:
    • Intraoperative Patients: Evaluate the agent in actual surgical patients to assess its performance under more challenging and varied conditions.
  • Complex Patient Populations: Explore the agent's capacity for patient-specific hypnosis in populations with less predictable PK/PD profiles (e.g., critically ill, morbidly obese).
  • Agent Improvement: Incorporate volunteer responses from this study into the training regimen to develop an even more refined agent.
  • Dual Agent System: Consider a system with two independent, cooperative agents to better balance the competing goals of induction speed and maintenance accuracy.

Summary updated

BibTeX
@inproceedings{2010reinforcement,
  title = {{Reinforcement Learning for Closed-Loop Propofol Anesthesia: A Human Volunteer Study}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2010},
}
Figure