Sequential Planning for Steering Immune System Adaptation

Christian Kroer, Tuomas Sandholm

Introduction

Biological systems are inherently dynamic and adaptive. While this adaptability is crucial for fighting infections, it also presents a significant challenge in medicine: diseases like cancer can develop resistance to drugs, and the immune system can mistakenly adapt to attack the body, leading to autoimmunity. Traditional medical treatments are often "static," involving a single drug or a fixed cocktail administered at once. However, biological signaling pathways are incredibly complex, characterized by intricate feedback loops that operate across various timescales.

This paper introduces a computational framework designed to "steer" the immune system's adaptation using sequential planning. By treating biological management as a sequential decision-making problem, the researchers demonstrate that multi-step, contingent treatment plans can significantly outperform traditional static therapies in guiding T cell differentiation.

Methodology: Modeling the Immune "Opponent"

The authors frame the challenge as a Partially Observable Markov Decision Process (POMDP). To solve this, they combine several advanced computational components:

  • Biological Logic Model: They utilize a calibrated Boolean logical network of a T cell. This model represents 59 different biological variables (proteins, receptors) as binary On/Off switches.
  • High-Fidelity Simulation: The BioNetGen software package serves as a "black box" simulator, allowing the planning agent to observe how the cell might react to specific stimuli over time.
  • Monte Carlo Tree Search (MCTS): The researchers employed the UCT (Upper Confidence Bound applied to Trees) variant of MCTS. Much like an AI playing Go, the planner explores different treatment sequences—activating or inhibiting cytokines like TCR, CD28, IL-2, and TGF-$β$—to find the path that maximizes a defined utility function.
The complex Boolean logical network of T cell signaling pathways used as the simulation environment.

To maintain computational tractability, the planner uses state abstraction, only "observing" two key proteins (FOXP3 and IL-2) to decide its next move, even though the underlying model tracks the full complexity of the signaling network shown above.

Key Insights and Results

The Power of Sequential Depth

The primary finding is that for complex biological goals, such as developing regulatory T cells (which suppress immune responses to prevent autoimmunity), sequential plans are vastly superior to static ones.

Expected Value (EV) across different simulation timesteps and planning depths for regulatory T cells. Increasing depth consistently improves utility.

As illustrated in the results above, increasing the planning depth from 1 to 2 or 3 led to a clear increase in Expected Value (EV) across all simulation timeframes. For example, at 360 timesteps, a 3-step plan nearly doubled the utility of a 1-step plan. This suggests that the planner is successfully navigating feedback loops that a single-step intervention cannot overcome.

Task-Dependent Complexity

Interestingly, the benefit of planning depth is task-dependent. While regulatory T cells required deep planning, the researchers found that for effector T cells (which fight cancer), a single-step plan was sufficient to reach maximum utility. This indicates that some biological states are "flatter" and easier to reach, while others require a more nuanced, multi-stage approach to bypass the system's natural regulatory checkpoints.

Exploiting Biological Pathways

Rather than assuming a "worst-case" game-theoretic opponent, the authors used an "opponent-exploitation" approach. By leveraging the simulator, the planner discovered non-obvious pathways through the signaling network. It successfully generated complex "if-then" treatment tables, such as initiating treatment with TGF-$β$ and IL-2 and then waiting to see if FOXP3 is expressed before deciding whether to trigger the TCR receptor.

Limitations and Future Directions

While promising, this work is currently limited to Boolean models, which simplify biological concentrations into binary states. Real-world systems are continuous, suggesting a need for models based on Ordinary Differential Equations (ODEs) in future iterations. Furthermore, these results are entirely in silico (computational). The next critical step is validating these sequential strategies in vitro (in the lab) and eventually in vivo (in living organisms) to account for the full complexity of a biological environment. Expanding the set of observable and manipulatable variables will also be necessary to refine these treatment policies for clinical use.

Summary updated

Abstract

Biological adaptation is a powerful mechanism that makes many disorders hard to combat. In this paper we study steering such adaptation through sequential planning. We propose a general approach where we leverage Monte Carlo tree search to compute a treatment plan, and the biological entity is modeled by a black-box simulator that the planner calls during planning. We show that the framework can be used to steer a biological entity modeled via a complex signaling pathway network that has numerous feedback loops that operate at different rates and have hard-to-understand aggregate behavior. We apply the framework to steering the adaptation of a patient's immune system. In particular, we apply it to a leading T cell simulator (available in the biological modeling package BioNetGen). We run experiments with two alternate goals: developing regulatory T cells or developing effector T cells. The former is important for preventing autoimmune diseases while the latter is associated with better survival rates in cancer patients. We are especially interested in the effect of sequential plans, an approach that has not been explored extensively in the biological literature. We show that for the development of regulatory cells, sequential plans yield significantly higher utility than the best static therapy. In contrast, for developing effector cells, we find that (at least for the given simulator, objective function, action possibilities, and measurement possibilities) single-step plans suffice for optimal treatment.
BibTeX
@inproceedings{kroer2016sequential,
  author = {Christian Kroer and Tuomas Sandholm},
  title = {{Sequential Planning for Steering Immune System Adaptation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure