Reinforcement learning for adaptive control of phenotypically heterogeneous bacterial populations

Josiah C. Kratz, Zhaoyang Wen, Shiladitya Banerjee, Oana CarjabioRxiv 2025 · DOI

TL;DR

This paper introduces a reinforcement learning (RL) framework that learns adaptive antibiotic treatment strategies for bacterial populations by only using population-level growth measurements. The RL agent can dynamically adjust both antibiotic dosage and nutrient availability, leading to more effective population control and even complete extinction, outperforming traditional fixed-dose or simple pulsing strategies.

What Problem Does This Paper Address?

Bacterial populations are remarkably resilient to antibiotics due to phenotypic heterogeneity – variations in how individual cells behave even within the same genetic makeup. This means that a single, fixed antibiotic treatment often fails, as some cells can survive and repopulate. Existing adaptive treatment strategies often require detailed knowledge of the bacterial system or hard-to-obtain measurements. This paper tackles the challenge of designing effective, adaptive treatments that can control these heterogeneous populations using only easily accessible, population-level data, and crucially, by considering nutrient availability as a controllable factor alongside antibiotics.

Methods

The core of this work is a model-free deep Q-learning (DQN) algorithm. This approach allows an "agent" to learn an optimal control policy through trial and error, interacting with a simulated bacterial population. The agent's goal is to minimize a cost function, which in this case is related to population growth rate, aiming for extinction.

Key methodological aspects include:

  • Virtual Bacterial Population Model: A sophisticated, multiscale computational model that simulates individual bacterial cells, their intracellular dynamics (proteome allocation, damage accumulation), and their response to environmental factors like nutrient availability and antibiotic concentration. This model captures both single-cell behavior and population-level outcomes.
  • State Representation: The agent's "state" is derived from population-level measurements, primarily the growth rate. To overcome partial observability (where growth rate doesn't fully reveal the underlying physiological state), the agent's input is augmented with a history of past actions and observations.
  • Action Space: The agent can control two key environmental factors: antibiotic concentration and nutrient availability.
  • Cost Function: The agent is trained to minimize the expected cumulative discounted cost, which is primarily defined by the instantaneous population growth rate. A large negative reward is given for population extinction.
  • Generalization Testing: The trained agents were evaluated in environments and conditions not encountered during training to assess their ability to generalize.

Datasets

The study uses a simulated virtual bacterial population as its "dataset." This model is based on previously published work [33] and incorporates detailed intracellular dynamics and population-level responses. The authors do not use real-world experimental datasets for training or evaluation in this paper.

Key Insights and Ideas

  • Nutrient Availability as a Crucial Control Lever: The study highlights that controlling nutrient availability, often overlooked in antibiotic treatment, is as important as antibiotic dosage for effective bacterial control.
  • Dual-Control Strategy for Extinction: By jointly controlling antibiotics and nutrients, the RL agent discovered a powerful strategy that consistently drives bacterial populations to extinction, a feat not achieved by drug-only adaptive strategies.
  • Inferring Hidden States: The RL agent can infer the underlying physiological state of the bacteria (e.g., stress protein levels) and even unobserved environmental fluctuations (like nutrient availability) solely from population-level growth rate measurements.
  • Generalization Beyond Training: The learned policies demonstrate remarkable generalization capabilities, performing well in environments and conditions not seen during training, suggesting they capture fundamental principles of bacterial susceptibility.
  • Mechanistic Insight from RL: The combination of RL with a mechanistic model allows for understanding why certain strategies are effective, providing valuable biological insights beyond just finding an optimal policy.

Key Results

  • Outperforming Fixed Strategies: The RL agent's learned policies consistently outperformed hard-coded antibiotic pulsing protocols in constant nutrient environments.
  • Achieving Population Extinction: When granted control over both antibiotics and nutrients, the RL agent achieved population extinction in 100% of simulations, a significant improvement over "Feast" (0% extinction) and "Famine" (sporadic extinction) protocols.
  • Robustness to Unobserved Fluctuations: The agent successfully inferred hidden nutrient dynamics and adapted its strategy to drive extinction even when nutrient levels were not directly measured.
  • Generalization to Unseen Environments: "Specialist" agents trained on single environments generalized well to unseen conditions, and a "generalist" agent trained on multiple environments also showed strong performance.
  • Limitations with Mutations: While robust to environmental changes, the learned policies showed degraded performance against mutating populations, especially at higher mutation rates, highlighting a key challenge for long-term control.

Limitations and Future Work

  • Mutation-Driven Resistance: The paper acknowledges that the learned policies degrade in effectiveness when bacterial populations accumulate mutations that alter their physiology and resistance. This is because mutations introduce non-stationarity, violating assumptions of the Q-learning algorithm.
  • Non-Stationary Environments: The authors suggest future work should focus on developing RL algorithms that can handle non-stationary environments more effectively, potentially using techniques like meta-learning or context-aware Markov decision processes.
  • Experimental Validation: While the study uses a sophisticated model, experimental validation of these RL-driven adaptive strategies in real bacterial systems is a crucial next step for clinical translation.
  • Contextual Information: Incorporating more contextual information into the RL algorithm could further improve its performance.

Summary updated

Abstract

Bacterial populations display extraordinary resilience to antibiotic stress, driven by diverse physiological states that allow some cells to persist and later repopulate. This phenotypic heterogeneity, amplified by environmental fluctuations, undermines the effectiveness of conventional fixed-dose treatment regimens. To address this challenge, we introduce a reinforcement learning (RL) framework that discovers adaptive treatment strategies using only experimentally accessible, population-level measurements. The RL agent learns to infer the hidden physiological state of the population and leverages this knowledge to maintain control even under conditions not encountered during training. Moreover, when granted control over nutrient availability, an important driver of physiological change often overlooked in antibiotic treatment protocols, the agent consistently drives population extinction, surpassing adaptive protocols based solely on drug dynamics. This computational framework offers a powerful, data-driven approach for designing adaptive treatment strategies to counter the growing threat of antimicrobial resistance.
BibTeX
@misc{kratz2025reinforcement,
  author = {Josiah C. Kratz and Zhaoyang Wen and Shiladitya Banerjee and Oana Carja},
  title = {{Reinforcement learning for adaptive control of phenotypically heterogeneous bacterial populations}},
  year = {2025},
  doi = {10.1101/2025.11.21.689767},
  howpublished = {bioRxiv},
}
Figure