Experimental Design via Generalized Mean Objective Cost of Uncertainty

Shahin Boluki, Xiaoning Qian, Edward R. DoughertyIEEE Access 2019 · DOI

TL;DR

This paper introduces a generalized framework for experimental design called Mean Objective Cost of Uncertainty (MOCU). This new framework unifies existing methods like Knowledge Gradient (KG) and Efficient Global Optimization (EGO) and offers greater flexibility in how we define uncertainty, actions, and experiments, leading to more efficient experimental design.

What Problem Does This Paper Address?

Designing effective experiments is crucial for scientific discovery and engineering. Often, we want to find the best "operator" (e.g., a filter, a material composition, a drug dosage) to achieve a specific objective. However, there's often uncertainty about the underlying system or model. Traditional approaches might design an operator optimal for a specific system, but this can be costly if the system deviates from that assumption. The Mean Objective Cost of Uncertainty (MOCU) quantifies this performance cost. The challenge this paper addresses is to generalize MOCU to be more flexible and to demonstrate how it encompasses and unifies various existing experimental design strategies, particularly those used in signal processing, genomics, and materials science.

Methods

The core contribution is the generalized MOCU framework. This framework defines:

  • Uncertainty Class ($\Theta$): The set of possible underlying system parameters.
  • Prior Distribution ($\pi$): The initial belief about these parameters.
  • Action Space ($\Psi$): The set of possible operators or decisions.
  • Cost Function ($C$): How well an action performs for a given system parameter.

The key innovation is the introduction of:

  • Intrinsically Bayesian Robust (IBR) action ($\psi_{IBR}$): An action that minimizes the expected cost over the entire uncertainty class. This is different from an action optimal for a specific system parameter.
  • Objective Cost of Uncertainty ($U_\psi$): The performance loss incurred by using the IBR action instead of the system-specific optimal action.
  • Mean Objective Cost of Uncertainty (MOCU): The average of the Objective Cost of Uncertainty over the prior distribution.

The paper then extends this to experimental design:

  • Experiment Space ($\Xi$): The set of possible experiments that can be performed.
  • Conditional Uncertainty Class ($\Theta|\xi$): The updated belief about system parameters after an experiment.
  • Experimental Design Value ($D_\psi(\Theta, \xi)$): The expected reduction in MOCU achieved by performing an experiment.

The optimal experiment is the one that maximizes this reduction. The paper shows how this generalized MOCU framework can be applied to sequential experiments and how it naturally leads to policies similar to Knowledge Gradient (KG) and Efficient Global Optimization (EGO) under specific assumptions.

Datasets

Not discussed in the paper. The paper focuses on theoretical formulation and simulation studies.

Key Insights and Ideas

  • Unification of Existing Methods: The generalized MOCU framework provides a single, elegant mathematical structure that encompasses and explains the underlying principles of established experimental design methods like KG and EGO. This is a significant conceptual advance.
  • Flexibility in Design: The generalized MOCU allows for more flexible definitions of the uncertainty class, action space, and experiment space. This is crucial for real-world problems where these elements might not neatly align or might be time-dependent.
  • Directly Addressing Model Uncertainty: Unlike KG, which often models uncertainty on the "reward" of an action, MOCU directly models uncertainty on the underlying physical system parameters. This can lead to more principled and effective designs, especially when the functional form of the model is known but its parameters are uncertain.
  • IBR Action as a Robust Baseline: The concept of an Intrinsically Bayesian Robust (IBR) action provides a clear baseline for comparison – it's the best we can do on average across all possibilities without any further experimentation. MOCU quantifies the cost of not having this perfect knowledge.
  • Sequential Design Advantage: The framework naturally extends to sequential experiments, where the goal is to iteratively reduce uncertainty. The paper shows how the MOCU reduction can guide stopping criteria for these iterative processes.

Key Results

  • The paper demonstrates that the generalized MOCU framework can reproduce the policies of Knowledge Gradient (KG) and Efficient Global Optimization (EGO) under their respective modeling assumptions.
  • A simulation study comparing MOCU-based design with KG for a quadratic function optimization problem showed that the MOCU-based policy consistently achieved a lower average opportunity cost, especially after the first few experimental iterations. This highlights the benefit of directly incorporating model parameter uncertainty.
  • The paper illustrates the application of MOCU-based experimental design in domains like materials discovery (using Ginzburg-Landau theory) and genetic networks, showing its broad applicability.

Limitations and Future Work

  • Computational Complexity: While the framework is general, applying it in practice can still be computationally intensive, especially when dealing with complex models or high-dimensional uncertainty spaces.
  • Prior Knowledge: The effectiveness of MOCU-based design relies on having a reasonable prior distribution for the uncertainty parameters. Eliciting accurate priors can be challenging.
  • Future Work: The authors suggest exploring finite-horizon optimal design using dynamic programming (as referenced in [8]) and further investigating the connection between MOCU and other Bayesian experimental design paradigms. The flexibility of the framework also opens doors for new applications in complex systems where direct modeling of uncertainty is paramount.

Summary updated

BibTeX
@article{boluki2019experimental,
  author = {Shahin Boluki and Xiaoning Qian and Edward R. Dougherty},
  title = {{Experimental Design via Generalized Mean Objective Cost of Uncertainty}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2018.2886576},
}
Figure