A Bayesian model of dose-response for cancer drug studies

Wesley Tansey, Christopher Tosh, David M. Blei

Abstract

Exploratory cancer drug studies test multiple tumor cell lines against multiple candidate drugs. The goal in each paired (cell line, drug) experiment is to map out the dose-response curve of the cell line as the dose level of the drug increases. We propose Bayesian Tensor Filtering (BTF), a hierarchical Bayesian model for dose-response modeling in multi-sample, multi-treatment cancer drug studies. BTF uses low-dimensional embeddings to share statistical strength between similar drugs and similar cell lines. Structured shrinkage priors in BTF encourage smoothness in the dose-response curves while remaining adaptive to sharp jumps when the data call for it. We focus on a pair of cancer drug studies exhibiting a particular pathology in their experimental design, leading us to a non-conjugate monotone mixture-of-Gammas likelihood. To perform posterior inference, we develop a variant of the elliptical slice sampling algorithm for sampling from linearly-constrained multivariate normal priors with non-conjugate likelihoods. In benchmarks, BTF outperforms state-of-the-art methods for covariance regression and dynamic Poisson matrix factorization. On the two cancer drug studies, BTF outperforms the current standard approach in biology and reveals potential new biomarkers of drug sensitivity in cancer.

Introduction

In oncology, high-throughput screening of tumor cell lines (organoids) against vast libraries of candidate drugs is essential for identifying potential therapies. These studies aim to characterize the dose-response curve—the relationship between drug concentration and cell survival. However, researchers face two significant challenges:

  1. Data Sparsity: Testing every drug against every cell line at every concentration is economically and logistically unfeasible, resulting in a sparse data tensor.
  2. Experimental Noise: Lab artifacts, such as "column effects" from multi-headed pipettes, introduce correlated errors that can skew results.

Furthermore, biological systems impose a monotonicity constraint: increasing drug dosage should not lead to increased cell survival. Standard Gaussian models often fail to incorporate these hard constraints or handle the non-standard noise distributions found in real-world lab data.

Methodology

The authors introduce Bayesian Tensor Filtering (BTF), a hierarchical framework that addresses these challenges through two primary innovations.

1. Bayesian Tensor Filtering (BTF)

BTF models the dose-response problem as a factorization task. Each cell line and drug is represented by a low-dimensional embedding, allowing the model to share "statistical strength" across similar entities.

  • Adaptive Smoothness: The model employs a Bayesian group trend filtering prior with horseshoe shrinkage. This encourages the dose-response curves to be smooth while remaining capable of capturing "sharp jumps"—sudden drops in survival that indicate drug efficacy.
  • Monotonicity Constraints: The biological requirement that survival must be non-increasing with dose is baked directly into the model's prior structure.

2. Generalized Analytic Slice Sampling (GASS)

The use of a non-conjugate mixture-of-Gammas likelihood and linear constraints makes traditional MCMC methods inefficient. To solve this, the authors developed GASS, an extension of elliptical slice sampling. GASS analytically determines the valid regions of the proposal ellipse defined by linear constraints, ensuring that every sample is valid and significantly improving the mixing speed of the Markov chain.

Results and Discussion

The BTF model was validated across four datasets, including a pilot study from Columbia University and a large-scale "Landscape" study from Samsung Medical Center.

Dose-Response Modeling

BTF demonstrates a superior ability to fit dose-response curves compared to standard methods like Non-negative Matrix Factorization (NMF). By enforcing monotonicity and sharing information across similar drugs, the model effectively filters out experimental noise.

Dose-response curve showing a strong sensitivity to the drug, where the model captures a sharp drop in survival while adhering to monotonicity constraints.

In cases where a drug has no effect on an organoid, the model correctly identifies a flat response, even when individual data points are noisy.

Dose-response curve for a non-responsive pair (Organoid 1, Drug 3), demonstrating the model's robustness to experimental noise.

Biomarker Discovery and Benchmarking

Beyond curve fitting, BTF's latent embeddings facilitate the discovery of genetic biomarkers. The model identified that EGFRvIII gene rearrangements predict sensitivity to Trametinib, and PTEN loss correlates with sensitivity to Neratinib—findings consistent with established cancer biology. In broader benchmarks, such as Google Flu Trends, BTF outperformed state-of-the-art covariance regression models (e.g., BNP-CovReg), achieving higher accuracy with fewer latent factors.

Conclusion

Bayesian Tensor Filtering provides a robust framework for analyzing complex, noisy, and sparse dose-response data. While the current MCMC-based implementation is computationally demanding, the integration of biological priors and the GASS algorithm represents a significant step forward in high-throughput drug screening. Future work may explore variational inference to scale the model to even larger datasets and the incorporation of explicit chemical features to predict responses for novel compounds.

Summary updated

Abstract

Exploratory cancer drug studies test multiple tumor cell lines against multiple candidate drugs. The goal in each paired (cell line, drug) experiment is to map out the dose-response curve of the cell line as the dose level of the drug increases. We propose Bayesian tensor filtering (BTF), a hierarchical Bayesian model for dose-response modeling in multisample, multitreatment cancer drug studies. BTF uses low-dimensional embeddings to share statistical strength between similar drugs and similar cell lines. Structured shrinkage priors in BTF encourage smoothness in the dose-response curves while remaining adaptive to sharp jumps when the data call for it. We focus on a pair of cancer drug studies exhibiting a particular pathology in their experimental design, leading us to a nonconjugate monotone mixture-of-gammas likelihood. To perform posterior inference, we develop a variant of the elliptical slice sampling algorithm for sampling from linearly-constrained multivariate normal priors with nonconjugate likelihoods. In benchmarks, BTF outperforms state-of-the-art methods for covariance regression and dynamic Poisson matrix factorization. On the two cancer drug studies, BTF outperforms the current standard approach in biology and reveals potential new biomarkers of drug sensitivity in cancer. Code is available at https://github.com/tansey/functionalmf.
BibTeX
@article{tansey2019bayesian,
  author = {Wesley Tansey and Christopher Tosh and David M. Blei},
  title = {{A Bayesian model of dose-response for cancer drug studies}},
  journal = {The Annals of Applied Statistics},
  year = {2019},
}
Figure