Bayesian Optimization with Dimension Scheduling: Application to Biological Systems

Doniyor Ulmasov, Caroline Baroukh, Benoît Chachuat, Marc Peter Deisenroth, Ruth MisenerarXiv 2015 · arXiv

Introduction

Bayesian Optimization (BO) is a powerful framework for optimizing expensive black-box functions. In computational biology, it is frequently used to estimate parameters for complex metabolic models. However, standard BO faces a significant computational bottleneck: the Gaussian Process (GP) surrogate models typically used have a computational cost that scales cubically ($O(n^3)$) with the number of observations. As biological simulations become faster and datasets grow larger, the optimizer itself can become slower than the model it aims to tune. Ulmasov et al. propose the Dimension Scheduling Algorithm (DSA) to overcome these limitations in high-dimensional biological systems.

The Dimension Scheduling Algorithm (DSA)

The core challenge addressed by DSA is maintaining the efficiency of BO as the number of parameters (dimensions) and data points increases. Instead of modeling the entire high-dimensional space with a single, computationally expensive GP, DSA employs several strategies to streamline the process:

  1. Dimension Subsets and Scheduling: At each iteration, DSA selects a small subset of dimensions to optimize while keeping others fixed at their best-known values.
  2. PCA-based Importance Sampling: To ensure the most impactful parameters are prioritized, the algorithm uses Principal Component Analysis (PCA) on the accumulated data. Dimensions corresponding to higher eigenvalues are scheduled for optimization more frequently.
  3. Distributed Gaussian Processes: Rather than one global GP, DSA utilizes multiple smaller GPs, each trained on data specific to a subset of dimensions. This significantly reduces the training overhead and keeps the $O(n^3)$ cost manageable.

Unlike previous methods like Additive BO, DSA does not assume the objective function is strictly decomposable, making it better suited for the highly interconnected parameter spaces found in biological systems.

Experimental Evaluation

The researchers evaluated DSA using ten different model variants of microalgae metabolism based on the DRUM framework. These models involve 10 to 12 dimensions, where the goal is to minimize the squared error between simulations and experimental data for biomass, lipids, and chlorophyll. DSA was benchmarked against standard BO over 500 iterations.

Results and Discussion

The experimental results demonstrate that DSA provides a substantial improvement in both computational efficiency and optimization performance.

Computational Efficiency

DSA achieved a dramatic reduction in total execution time compared to standard BO. By utilizing smaller, distributed GPs and optimizing subsets of dimensions, the mathematical overhead of the optimizer was slashed.

Average running time comparison between BO and DSA across various metabolic models.

As shown in the figure above, DSA consistently completed optimization tasks in approximately one-fifth of the time required by traditional BO across all tested models (M19 through M35).

Optimization Performance

Beyond speed, DSA also proved more effective at finding lower objective values. Standard BO often struggles in high-dimensional spaces as the GP becomes cumbersome and exploration slows down. In contrast, DSA's "slicing" approach allows it to explore the parameter landscape more rapidly.

Convergence of objective values over 500 iterations for DSA and standard BO on selected models.

The convergence plots indicate that DSA (solid lines) typically reaches lower objective values faster than standard BO (dashed/dotted lines). In nearly all tested microalgae models, DSA outperformed the benchmark within the same iteration limit, demonstrating its robustness for complex biological parameter estimation.

Conclusion

The Dimension Scheduling Algorithm (DSA) effectively addresses the scalability issues of Bayesian Optimization in high-dimensional biological contexts. By focusing on impactful dimension subsets and reducing GP complexity, DSA offers a 5x speedup and superior accuracy over standard methods. Future work may explore hybrid approaches that combine DSA's rapid exploration with gradient-based local refinement to further polish final results.

Summary updated

Abstract

Bayesian Optimization (BO) is a data-efficient method for global black-box optimization of an expensive-to-evaluate fitness function. BO typically assumes that computation cost of BO is cheap, but experiments are time consuming or costly. In practice, this allows us to optimize ten or fewer critical parameters in up to 1,000 experiments. But experiments may be less expensive than BO methods assume: In some simulation models, we may be able to conduct multiple thousands of experiments in a few hours, and the computational burden of BO is no longer negligible compared to experimentation time. To address this challenge we introduce a new Dimension Scheduling Algorithm (DSA), which reduces the computational burden of BO for many experiments. The key idea is that DSA optimizes the fitness function only along a small set of dimensions at each iteration. This DSA strategy (1) reduces the necessary computation time, (2) finds good solutions faster than the traditional BO method, and (3) can be parallelized straightforwardly. We evaluate the DSA in the context of optimizing parameters of dynamic models of microalgae metabolism and show faster convergence than traditional BO.
BibTeX
@misc{ulmasov2015bayesian,
  author = {Doniyor Ulmasov and Caroline Baroukh and Benoît Chachuat and Marc Peter Deisenroth and Ruth Misener},
  title = {{Bayesian Optimization with Dimension Scheduling: Application to Biological Systems}},
  year = {2015},
  eprint = {1511.05385},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.05385},
}
Figure