Deep Reinforcement Learning for Optimal Experimental Design in Biology

Neythen J. Treloar, Nathan Braniff, Brian Ingalls, C. BarnesBiorxiv 2022 · DOI

TL;DR

This paper introduces a novel approach to optimal experimental design (OED) in biology using deep reinforcement learning (RL). By framing OED as an RL problem, the authors demonstrate that their method can design experiments that are more informative for parameter estimation than traditional methods, even when faced with uncertainty about the system's true parameters.

What Problem Does This Paper Address?

Understanding complex biological systems often relies on building accurate mechanistic models. However, these models are typically non-linear and noisy, making parameter estimation a challenging task. Furthermore, experiments to gather data for parameter inference can be resource-intensive and time-consuming. Optimal Experimental Design (OED) aims to maximize the information gained from experiments, thereby improving parameter estimates and reducing experimental effort. Traditional OED methods often struggle with non-linear systems, require prior knowledge of parameter values, or involve computationally expensive calculations. This paper addresses the challenge of designing experiments for non-linear biological systems, particularly when there is uncertainty about the system's parameters, by leveraging the power of reinforcement learning.

Methods

The core of this work is the application of deep reinforcement learning to OED. The authors frame the problem as an RL agent learning a policy to select experimental inputs that maximize the determinant of the Fisher Information Matrix (FIM), a measure of how well parameters can be estimated.

  • Initial Approach (FQ-learning): They first explore a Fitted Q-learning (FQ) algorithm. This approach, while showing promise, still relies on prior knowledge of the system's parameters to calculate the FIM and thus the reward.
  • Recurrent T3D Algorithm: To overcome the reliance on prior parameter knowledge, they develop a more advanced algorithm called Recurrent Twin Delayed Deep Deterministic Policy Gradient (RT3D). This algorithm uses a recurrent neural network to process the entire history of experimental observations and inputs, allowing the agent to infer its position in parameter space and adapt its experimental design accordingly. This approach enables the agent to learn optimal experimental designs without explicit knowledge of the true parameter values.
  • Training over Parameter Distributions: A key innovation is training the RL agent not on a single set of parameters, but on a distribution of possible parameter values. This allows the trained agent to generalize and design experiments that are robust to parametric uncertainty.

Datasets

The authors use a simulated chemostat model of auxotrophic bacterial growth as their primary experimental system. This model is described by a set of differential equations, and the parameters of this model are varied for training and evaluation. No new benchmark datasets were introduced; rather, the authors created simulated data based on their model.

Key Insights and Ideas

  • RL as a Powerful OED Tool: Reinforcement learning, particularly the RT3D algorithm, can effectively learn to design experiments that maximize information gain, outperforming traditional methods in certain aspects.
  • Decoupling from Prior Knowledge: The RT3D algorithm's ability to learn from the history of observations and inputs, and to be trained over parameter distributions, significantly reduces the need for precise prior knowledge of system parameters. This is a major step towards more practical and automated experimental design.
  • Robustness to Uncertainty: Training the RL agent over a distribution of parameters makes it robust to parametric uncertainty, a common challenge in biological systems. The agent learns a general strategy that performs well across a range of parameter values.
  • Complementary to Existing Methods: The RL approach is presented as a complementary tool to existing OED methods, offering a way to handle complex non-linear systems and parameter uncertainty.

Key Results

  • The RT3D algorithm achieved an average optimality score of 19.84, comparable to or exceeding that of a Model Predictive Controller (MPC) which had full knowledge of the parameters.
  • The best performing RT3D agent outperformed a one-step ahead optimizer (OSAO) and a basic FQ-learning agent in terms of optimality score.
  • When trained over a parameter distribution, the RT3D controller performed nearly as well as an MPC controller with complete parameter knowledge, demonstrating its robustness to uncertainty.
  • The RT3D algorithm showed significantly improved performance over the FQ algorithm with lower training costs.
  • Parameter estimation accuracy (measured by normalized MSE and log covariance) correlated well with the D-optimality scores achieved by the experimental designs.

Limitations and Future Work

  • Training Time: While the RT3D algorithm is efficient in terms of data and computation time compared to some RL approaches, the training process can still be lengthy (around 12-15 hours).
  • Model Structure Assumption: The approach still relies on the assumption that the underlying model structure is accurate.
  • Future Work: The authors suggest extending the approach to other optimality criteria (e.g., A-optimality, E-optimality), training over more complex parameter distributions (non-symmetric, multi-modal), and incorporating model selection and divergence of model predictions into the reward function. They also highlight the potential for applying this method to other scientific and engineering domains.

Summary updated

Abstract

Abstract The field of optimal experimental design uses mathematical techniques to determine experiments that are maximally informative from a given experimental setup. Here we apply a technique from artificial intelligence—reinforcement learning—to the optimal experimental design task of maximizing confidence in estimates of model parameter values. We show that a reinforcement learning approach performs favourably in comparison with a one-step ahead optimisation algorithm and a model predictive controller for the inference of bacterial growth parameters in a simulated chemostat. Further, we demonstrate the ability of reinforcement learning to train over a distribution of parameters, indicating that this approach is robust to parametric uncertainty. 1 Author summary Biological systems are often complex and typically exhibit non-linear behaviour, making accurate model parametrisation difficult. Optimal experimental design tools help address this problem by identifying experiments that are predicted to provide maximally accurate parameter estimates. In this work we use reinforcement learning, an artificial intelligence method, to determine such experiments. Our simulation studies show that this approach allows uncertainty in model parameterisation to be directly incorporated into the search for optimal experiments, opening a practical avenue for training an experimental controller without confident knowledge of the system’s parameter values. We present this method as complementary to existing optimisation approaches and we anticipate that artificial intelligence has a fundamental role to play in the future of optimal experimental design.
BibTeX
@misc{treloar2022deep,
  author = {Neythen J. Treloar and Nathan Braniff and Brian Ingalls and C. Barnes},
  title = {{Deep Reinforcement Learning for Optimal Experimental Design in Biology}},
  year = {2022},
  doi = {10.1101/2022.05.09.491138},
  howpublished = {Biorxiv},
}
Figure