Interpreting tree ensemble machine learning models with endoR

Albane Ruaud, Niklas Pfister, Ruth E. Ley, Nicholas D. Youngblut

TL;DR

This paper introduces endoR, a novel R package designed to interpret complex "black-box" tree ensemble machine learning models, particularly in the context of microbiome data. endoR simplifies these models into a more understandable "decision ensemble," revealing feature importance, pairwise interactions, and visualizing them as an interpretable network, thereby enabling deeper biological insights.

What Problem Does This Paper Address?

Machine learning models, especially tree ensembles like Random Forests, are powerful tools for predicting phenotypes from high-dimensional, sparse microbiome data. However, their "black-box" nature makes it difficult to understand why a prediction is made. This lack of interpretability limits our ability to gain biological insights into the complex relationships between the microbiome and host health or disease. Existing methods for model interpretation often struggle with the scale and complexity of microbiome data, providing only limited insights into feature interactions or being computationally expensive.

Methods

endoR tackles this by:

  • Simplifying the Model: It first extracts the core "rules" or "decisions" from a fitted tree ensemble model and then simplifies these into a "decision ensemble." This process involves regularization techniques like pruning and stability selection to reduce noise and retain only the most robust decisions.
  • Extracting Importance and Influence: For individual features and pairs of features (interactions), endoR calculates their "importance" (how much they improve prediction accuracy) and "influence" (how they affect model predictions).
  • Visualizing as a Network: The extracted information is visualized as a "decision network" where nodes represent features and edges represent interactions. Node size and color indicate feature importance and influence, while edge width and color represent interaction importance and influence. This network provides an intuitive overview of the model's logic.
  • Robustness and Efficiency: endoR incorporates adjustable regularization and bootstrapping to ensure stability and control for false discoveries. It is also designed to be computationally efficient, outperforming other methods like SHAP in speed.

Datasets

The authors evaluated endoR on:

  • Simulated Datasets:
    • Fully Simulated Data (FSD): Datasets where both features and the response variable were synthetically generated, allowing for a controlled evaluation of endoR's ability to recover known relationships.
  • Artificial Phenotypes (AP): Datasets using real metagenomic measurements but with artificially generated response variables. This benchmark is designed to mimic the complexity of real microbiome data while providing a known ground truth for evaluation.
  • Real Metagenome Datasets:
    • Cirrhosis Dataset: A previously published dataset of gut microbiomes from individuals with and without cirrhosis, used to validate endoR's ability to rediscover known associations and uncover new ones.
  • Methanobacteriaceae Dataset: A large, global dataset of 2203 individuals used to investigate factors influencing the presence of gut methanogens.

Key Insights and Ideas

  • Bridging the Gap: endoR effectively bridges the gap between powerful but opaque tree ensemble models and the need for interpretable biological insights in microbiome research.
  • Beyond Single Features: The ability to visualize and quantify pairwise interactions is crucial for understanding complex biological systems like the gut microbiome, where taxa rarely act in isolation.
  • Network as a Biological Hypothesis Generator: The decision network visualization provides a clear, interpretable map of how microbial features and their interactions contribute to a predicted outcome, directly facilitating the generation of novel hypotheses.
  • Robustness to Model Quality: While endoR performs best with accurate input models, its own internal regularization makes it robust even with less precise predictive models.

Key Results

  • Comparable or Better Accuracy: endoR inferred true associations with comparable or better accuracy than commonly used approaches on simulated and real data.
  • Enhanced Interpretability: endoR's visualizations and summary outputs significantly ease model interpretation compared to methods like SHAP, especially for high-dimensional microbiome data.
  • Confirmed Previous Findings: endoR successfully rediscovered known associations, such as gut microbiome differences in cirrhosis patients and the link between Methanobacteriaceae and Christensenellales.
  • Novel Discoveries: The tool revealed new associations, like Methanobacteriaceae being linked to a network of hydrogen-producing bacteria, and identified specific taxa (e.g., Megasphaera micronuciformis) as important markers for cirrhosis.
  • Computational Efficiency: endoR was substantially faster than SHAP for analyzing Random Forest models, scaling linearly with dataset dimensionality and sample size.

Limitations and Future Work

  • Model Dependence: The performance of endoR is dependent on the quality of the input tree ensemble model. A poor predictive model will lead to misleading interpretations.
  • Hyperparameter Tuning: While endoR provides sensible defaults, careful consideration of parameters like a (controlling false discoveries) and B (number of bootstraps) is recommended based on the authors' extensive evaluations.
  • Future Work: The authors suggest that endoR can be applied to other complex systems beyond microbiome data. The framework's flexibility allows for its use with various tree-based models and feature types. The development of more sophisticated regularization and visualization techniques could further enhance its utility.

Summary updated

Abstract

Background Tree ensemble machine learning models are increasingly used in microbiome science as they are compatible with the compositional, high-dimensional, and sparse structure of sequence-based microbiome data. While such models are often good at predicting phenotypes based on microbiome data, they only yield limited insights into how microbial taxa or genomic content may be associated. Results: We developed endoR, a method to interpret a fitted tree ensemble model. First, endoR simplifies the fitted model into a decision ensemble from which it then extracts information on the importance of individual features and their pairwise interactions and also visualizes these data as an interpretable network. Both the network and importance scores derived from endoR provide insights into how features, and interactions between them, contribute to the predictive performance of the fitted model. Adjustable regularization and bootstrapping help reduce the complexity and ensure that only essential parts of the model are retained. We assessed the performance of endoR on both simulated and real metagenomic data. We found endoR to infer true associations with more or comparable accuracy than other commonly used approaches while easing and enhancing model interpretation. Using endoR, we also confirmed published results on gut microbiome differences between cirrhotic and healthy individuals. Finally, we utilized endoR to gain insights into components of the microbiome that predict the presence of human gut methanogens, as these hydrogen-consumers are expected to interact with fermenting bacteria in a complex syntrophic network. Specifically, we analyzed a global metagenome dataset of 2203 individuals and confirmed the previously reported association between Methanobacteriaceae and Christensenellales . Additionally, we observed that Methanobacteriaceae are associated with a network of hydrogen-producing bacteria. Conclusion: Our method accurately captures how tree ensembles use features and interactions between them to predict a response. As demonstrated by our applications, the resultant visualizations and summary outputs facilitate model interpretation and enable the generation of novel hypotheses about complex systems. An implementation of endoR is available as an open-source R-package on GitHub ( https://github.com/leylabmpi/endoR ).
BibTeX
@article{ruaud2022interpreting,
  author = {Albane Ruaud and Niklas Pfister and Ruth E. Ley and Nicholas D. Youngblut},
  title = {{Interpreting tree ensemble machine learning models with endoR}},
  journal = {PLOS Computational Biology},
  year = {2022},
  doi = {10.1371/journal.pcbi.1010714},
}
Figure