A reinforced learning approach to optimal design under model uncertainty

Mingyao Ai, Holger Dette, Zhengfu Liu, Jun YuarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces a reinforcement learning framework to solve a classic "chicken-and-egg" problem in statistics: you need a model to design an optimal experiment, but you need experimental data to find the right model. By treating candidate models as "arms" in a multi-armed bandit and using Thompson Sampling, the authors develop a sequential strategy that identifies the true model and achieves the same efficiency as if the model were known from the start.

What Problem Does This Paper Address?

In experimental design, "optimal" usually means choosing data points that minimize the variance of your parameter estimates. However, these designs are notoriously sensitive to model specification. If you design an experiment for a linear relationship but the true process is exponential, your data might be collected at points that provide almost no information about the actual curve.

Researchers usually face a trade-off:

  1. Model Discrimination: Designing an experiment to figure out which model is correct.
  2. Parameter Estimation: Designing an experiment to get precise values for a specific model.

These two goals often require very different sets of experimental conditions. Traditional approaches like "robust designs" or "two-stage designs" often end up being "jack of all trades, master of none"—they are sub-optimal for the true model because they spend too much "budget" hedging against incorrect ones.

Methods

The authors frame the problem as a Reinforcement Learning (RL) task, specifically using a Multi-Armed Bandit approach.

The Algorithm: Multi-stage $\mathcal{M}$-Optimal Design

The experiment is broken into $T$ stages. In each stage:

  • Selection (Thompson Sampling): Instead of just picking the model that looks best so far, the algorithm samples from a posterior distribution (a Beta distribution). This allows it to "exploit" the most likely model while still "exploring" others.
  • Hybrid Design: The algorithm doesn't just use the optimal design for the selected model. It uses a hybrid: a mix of the model-specific optimal design and a uniform design. The uniform part ensures that the data collected is still useful for checking the goodness-of-fit of all candidate models.
  • Evaluation & Updating: After collecting data in a stage, the models are scored using the Bayesian Information Criterion (BIC). The "winning" model gets a "success" point, updating the Beta distribution for the next stage's Thompson Sampling.

Datasets

The paper evaluates the methodology using three distinct simulation studies rather than static datasets:

  1. Dose-Response Studies: Comparing Emax, linear, and exponential models (common in clinical trials).
  2. Robust Parameter Design: A scenario with three control factors and three noise factors to see if the algorithm can identify which noise factors actually impact the response.
  3. Multivariate Predictors: Linear regressions with six candidate models of varying complexity to test the algorithm's ability to handle aliasing and model estimability issues.

Key Insights and Ideas

  • Asymptotic Optimality: The authors prove that as the number of stages increases, the efficiency of their design converges to 1. This means that, in the long run, you lose nothing by not knowing the true model at the start.
  • Logarithmic Regret: The "cost" of uncertainty (the number of times a sub-optimal design is chosen) grows only at a rate of $O(\log T)$. This is the theoretical "gold standard" for efficiency in reinforcement learning.
  • Computational Efficiency: Unlike "compound" optimality criteria—which require solving a massive, multi-objective optimization problem—this RL approach only requires calculating optimal designs for individual models, which is much simpler and faster.
  • Information Recycling: By using a hybrid design with a uniform component, the algorithm ensures that even "failed" stages (where the wrong model was picked) still provide valuable data for model discrimination.

Key Results

  • Superior Efficiency: In dose-response simulations, the RL approach (Algorithm 1) consistently achieved higher D-efficiency than standard uniform, robust, or hybrid designs.
  • Rapid Convergence: In the robust parameter design study, the algorithm's efficiency reached approximately 97% by the time 256 samples were collected, significantly outperforming static robust designs.
  • Cost Savings: The RL approach was shown to require a substantially smaller budget to reach high accuracy because it stops controlling for "redundant" noise factors once it identifies they aren't part of the true model.

Limitations and Future Work

  • Locally Optimal Designs: For non-linear models, the algorithm currently requires "locally optimal" designs, which assume you have a rough initial guess of the parameters. If the initial guess is very far off, the efficiency could suffer.
  • Linearity Assumptions: The theoretical proofs rely on certain stability conditions (Condition 1 and 2) regarding the model selection criterion (BIC).
  • Future Directions: The authors suggest extending this framework to Bayes-maximin criteria, which would make the design even more robust to the initial parameter guesses. They also suggest exploring different model evaluation strategies to improve the statistical power of the discrimination step.

Summary updated

Abstract

Optimal designs are usually model-dependent and likely to be sub-optimal if the postulated model is not correctly specified. In practice, it is common that a researcher has a list of candidate models at hand and a design has to be found that is efficient for selecting the true model among the competing candidates and is also efficient (optimal, if possible) for estimating the parameters of the true model. In this article, we use a reinforced learning approach to address this problem. We develop a sequential algorithm, which generates a sequence of designs which have asymptotically, as the number of stages increases, the same efficiency for estimating the parameters in the true model as an optimal design if the true model would have correctly been specified in advance. A lower bound is established to quantify the relative efficiency between such a design and an optimal design for the true model in finite stages. Moreover, the resulting designs are also efficient for discriminating between the true model and other rival models from the candidate list. Some connections with other state-of-the-art algorithms for model discrimination and parameter estimation are discussed and the methodology is illustrated by a small simulation study.
BibTeX
@misc{ai2023reinforced,
  author = {Mingyao Ai and Holger Dette and Zhengfu Liu and Jun Yu},
  title = {{A reinforced learning approach to optimal design under model uncertainty}},
  year = {2023},
  eprint = {2303.15887},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.15887},
}

Loading related content...

Figure