Probabilistic learning of treatment trees in cancer

Tsung-Hung Yao, Zhenke Wu, Karthik Bharath, Jinju Li, Veerabhadran BaladandayuthapaniThe Annals of Applied Statistics 2022

Abstract

Accurate identification of synergistic treatment combinations and their underlying biological mechanisms is critical across many disease domains, especially cancer. In translational oncology research, preclinical systems such as patient-derived xenografts (PDX) have emerged as a unique study design evaluating multiple treatments administered to samples from the same human tumor implanted into genetically identical mice. In this paper, we propose a novel Bayesian probabilistic tree-based framework for PDX data to investigate the hierarchical relationships between treatments by inferring treatment cluster trees, referred to as treatment trees (Rx-tree). The framework motivates a new metric of mechanistic similarity between two or more treatments accounting for inherent uncertainty in tree estimation; treatments with a high estimated similarity have potentially high mechanistic synergy. Building upon Dirichlet Diffusion Trees, we derive a closed-form marginal likelihood encoding the tree structure, which facilitates computationally efficient posterior inference via a new two-stage algorithm. Simulation studies demonstrate superior performance of the proposed method in recovering the tree structure and treatment similarities. Our analyses of a recently collated PDX dataset produce treatment similarity estimates that show a high degree of concordance with known biological mechanisms across treatments in five different cancers. More importantly, we uncover new and potentially effective combination therapies that confer synergistic regulation of specific downstream biological pathways for future clinical investigations. Our accompanying code, data, and shiny application for visualization of results are available at: https://github.com/bayesrx/RxTree.

Introduction

In cancer research, "combination therapy"—using multiple drugs at once—is essential for overcoming drug resistance. However, testing every possible drug combination in clinical trials is logistically impossible. Researchers use Patient-Derived Xenografts (PDX)—where human tumor fragments are grown in mice—to test many treatments on the same tumor "avatar."

The challenge lies in analyzing this data. Standard methods like Pearson correlation or basic hierarchical clustering often fail to account for the inherent noise in biological data or the uncertainty of the tree structure itself. Furthermore, these methods don't easily reveal why certain drugs work well together. There is a critical need for a model that can infer the underlying biological mechanisms (the "how") from the observed tumor responses (the "what").

Conceptual overview of the Rx-tree framework mapping treatment responses to hierarchical mechanistic clusters.

Methods

The authors introduce Rx-tree, a Bayesian framework built on Dirichlet Diffusion Trees (DDT). In this model, treatments are represented as leaves on a tree, where branching points (internal nodes) represent unobserved biological mechanisms. Treatments that stay clustered together longer on the tree are inferred to share more similar mechanisms.

A major technical contribution is the derivation of a closed-form marginal likelihood for the tree structure. By mathematically "integrating out" the unobserved internal nodes, the model becomes significantly more computationally efficient.

The Dirichlet Diffusion Tree (DDT) model (A) and the resulting covariance matrix (B) used for efficient marginal likelihood calculation.

Inference is performed via a two-stage algorithm:

  1. Stage 1: Uses Approximate Bayesian Computation (ABC) to estimate global parameters like noise levels and divergence rates.
  2. Stage 2: Uses a Metropolis-Hastings (MH) algorithm to sample the posterior distribution of the tree structures.

To quantify similarity, the authors introduce the integrated Posterior Co-clustering Probability (iPCP). This metric averages across thousands of likely trees to provide a robust measure of mechanistic similarity, accounting for structural uncertainty.

Results

The framework was evaluated using both simulated datasets and the NIBR-PDXE dataset, which contains responses from over 1,000 PDX lines across five cancers (Breast, Melanoma, Colorectal, Lung, and Pancreatic).

Key findings include:

  • Superior Accuracy: Rx-tree outperformed standard hierarchical clustering and Pearson correlation in recovering true treatment relationships in simulations.
  • Biological Concordance: The model successfully grouped drugs with known shared targets (e.g., PI3K pathway inhibitors) without prior knowledge of chemical structures, validating that response data contains a "mechanistic signature."
  • New Combination Leads: The model uncovered promising new synergistic pairs, such as PI3K-ERBB3 for colorectal cancer and BRAF-PI3K for melanoma, which showed synergistic regulation of downstream pathways.

Conclusion and Future Work

The Rx-tree framework provides a powerful probabilistic approach to mapping treatment mechanisms. While currently assuming patient independence and binary branching, future extensions could incorporate patient-specific genomic data (multi-omics) to refine results. This would potentially allow for "precision" trees tailored to specific patient subtypes, further advancing the discovery of effective combination therapies.

Summary updated

Abstract

Accurate identification of synergistic treatment combinations and their underlying biological mechanisms is critical across many disease domains, especially cancer. In translational oncology research, preclinical systems such as patient-derived xenografts (PDX) have emerged as a unique study design evaluating multiple treatments administered to samples from the same human tumor implanted into genetically identical mice. In this paper, we propose a novel Bayesian probabilistic tree-based framework for PDX data to investigate the hierarchical relationships between treatments by inferring treatment cluster trees, referred to as treatment trees (R<sub>x</sub>-tree). The framework motivates a new metric of mechanistic similarity between two or more treatments accounting for inherent uncertainty in tree estimation; treatments with a high estimated similarity have potentially high mechanistic synergy. Building upon Dirichlet Diffusion Trees, we derive a closed-form marginal likelihood encoding the tree structure, which facilitates computationally efficient posterior inference via a new two-stage algorithm. Simulation studies demonstrate superior performance of the proposed method in recovering the tree structure and treatment similarities. Our analyses of a recently collated PDX dataset produce treatment similarity estimates that show a high degree of concordance with known biological mechanisms across treatments in five different cancers. More importantly, we uncover new and potentially effective combination therapies that confer synergistic regulation of specific downstream biological pathways for future clinical investigations. Our accompanying code, data, and shiny application for visualization of results are available at: https://github.com/bayesrx/RxTree.
BibTeX
@article{yao2022probabilistic,
  author = {Tsung-Hung Yao and Zhenke Wu and Karthik Bharath and Jinju Li and Veerabhadran Baladandayuthapani},
  title = {{Probabilistic learning of treatment trees in cancer}},
  journal = {The Annals of Applied Statistics},
  year = {2022},
}
Figure