The Unreasonable Effectiveness of Inverse Reinforcement Learning in Advancing Cancer Research

John Kalantari, Heidi Nelson, Nicholas ChiaAAAI 2020 · DOI

Introduction

Cancer biology is currently "data-rich but insight-poor." Despite massive amounts of genomic sequencing data, traditional statistical methods often struggle to capture the underlying process of cancer. Most models treat mutations as independent events, ignoring that cancer is a sequential, evolutionary "game" where cells mutate to navigate host barriers and outcompete neighbors.

This research introduces PUR-IRL, a Bayesian nonparametric framework that treats cancer progression as an "optimal" evolutionary strategy. By applying Inverse Reinforcement Learning (IRL) to genomic data, the authors successfully reverse-engineered the "reward functions" driving tumor growth, effectively rediscovering established clinical models of colorectal cancer using data from only 27 patients.

Methods: Cancer as a Markov Decision Process

The researchers propose a fundamental shift: viewing cancer as an agent in a Markov Decision Process (MDP). In this framework, States represent the genetic profile and latent biological features of a cell, Actions are specific somatic mutations (e.g., in the KRAS gene), and the Policy is the strategy the cancer uses to choose its next mutation.

To decode this strategy, the authors developed the PUR-IRL (Pop-Up Restaurant for IRL) algorithm. Unlike standard IRL, which assumes a single reward function, PUR-IRL uses a Bayesian nonparametric prior to infer an unbounded number of reward functions, accounting for different cancer subtypes and handling non-uniform sampling.

The PUR-IRL framework integrates a stochastic dynamics model with a Bayesian nonparametric algorithm to infer reward functions from observed mutational paths.

Results and Biological Validation

The approach was validated using simulated GridWorld data and Whole Genome Sequencing (WGS) from 27 colorectal cancer (CRC) patients. From the patient data, 215,000 possible evolutionary paths were extracted using phylogenetic reconstruction tools.

The reward functions inferred by PUR-IRL highlighted specific state-action pairs that correspond to the "Vogelgram," the gold-standard model of colorectal cancer progression (APC → KRAS → SMAD4 → TP53). By assuming the data has a logical, sequential structure, the model gained more insight from 27 patients than frequency-based statistics often gain from thousands.

Analysis of colorectal cancer data: (A) Heatmap of traversed paths, (B) inferred mutational sequences, and (C) the canonical Vogelgram model of progression.

On the GridWorld benchmark, PUR-IRL achieved an F1-score of 0.9328, demonstrating robustness even under biased sampling conditions. In the CRC dataset, it successfully identified the canonical sequence of mutations that characterizes disease progression, outperforming previous nonparametric IRL methods like DPM-BIRL.

Conclusion and Future Directions

The "unreasonable effectiveness" of this approach lies in its structural assumptions. By treating successful tumors as "winners" of an evolutionary game, the model can infer the underlying logic of progression.

Future work aims to scale this to more complex or rare cancers, which will require further refinement of feature engineering. Additionally, incorporating non-genomic data—such as clinical records, environmental factors, and proteomics—could create a more holistic "reward function" for cancer, potentially leading to better predictive models for patient outcomes.

Summary updated

BibTeX
@inproceedings{kalantari2020unreasonable,
  author = {John Kalantari and Heidi Nelson and Nicholas Chia},
  title = {{The Unreasonable Effectiveness of Inverse Reinforcement Learning in Advancing Cancer Research}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5380},
}
Figure