Problems with the nested granularity of feature domains in bioinformatics: the eXtasy case

Dusan Popovic, Alejandro Sifrim, Jesse Davis, Yves Moreau, Bart De MoorBMC Bioinformatics 2015 · DOI

Introduction

In bioinformatics, predictive models often struggle with data that is not "flat" but organized in a hierarchy. A single patient may have multiple chromosomes, each chromosome contains many genes, and each gene can harbor numerous mutations. This structural complexity is known as nested granularity.

Standard machine learning algorithms, such as Random Forests, often fail to account for this hierarchy. Instead of learning generalizable biological rules, they may inadvertently "cheat" by memorizing group-level patterns (like a specific gene's ID or its unique haploinsufficiency score) as a proxy for the target outcome. This leads to over-optimistic performance estimates during training that fail to translate into real-world clinical utility.

Example of nested granularity in the eXtasy dataset, where multiple variants share identical gene-level features like haploinsufficiency scores.

The "Shortcut" Trap

When a model is trained on nested data without proper controls, it often falls into a "shortcut" trap. For instance, if a specific gene in the training set has a unique score and ten different mutations, a standard model might learn to recognize that score as a lookup table for that gene's identity. When tested on an unseen gene, the model's performance plummets because it never learned the underlying features of deleterious mutations—only how to recognize the "neighborhoods" (genes) it saw during training.

Methods: Hierarchical Sampling

To address this, the authors propose a modification to the standard Random Forest algorithm called Hierarchical Sampling.

While standard Random Forests use "bootstrapping" (randomly picking samples from the entire dataset with replacement), the hierarchical approach first groups the data by its coarsest level (e.g., Gene ID). For every tree in the forest, the algorithm selects exactly one instance from each group. By ensuring that a single tree never sees multiple mutations from the same gene, the model is effectively "blinded" to group-level identifiers. This forces the forest to find patterns in finer-grained, mutation-level features that are more likely to generalize to unseen data.

Experimental Evaluation

The study evaluated this method using two distinct approaches:

  1. Synthetic Data: A controlled environment with 4,000 examples allowed the authors to manually tune "grain size" (the number of instances per group) and label noise. This helped identify exactly where standard models begin to overfit on non-informative group features.
  2. The eXtasy Benchmark: A real-world dataset for variant prioritization consisting of 24,454 disease-causing variants from the Human Gene Mutation Database (HGMD) and rare mutations from healthy individuals as controls.
Synthetic data distributions used to evaluate model performance under varying levels of feature informativeness and noise.

Key Results

  • Robustness to Grain Size: In synthetic tests, as the number of mutations per gene increased, the accuracy of standard Random Forests dropped significantly due to overfitting. The hierarchical sampling model remained stable.
  • Improved Precision in eXtasy: When applied to realistic, imbalanced biological data, the hierarchical model showed a significant jump in Precision (from 0.71 to 0.84).
  • AUC-PR Gains: The hierarchical approach outperformed the standard approach in Area Under the Precision-Recall Curve (0.128 vs 0.095), demonstrating a superior ability to filter out false positives in the high-confidence regions researchers care about most.

Conclusion

Hierarchical sampling offers a more parsimonious and robust approach to bioinformatics modeling. While it requires researchers to manually identify the "coarsest grain" of their data, the trade-off is a model that prioritizes reliable, generalizable biological insights over over-optimistic training metrics. Future work will focus on integrating this technique into the core eXtasy framework and adapting it for other ensemble classifiers like Boosted Trees.

Summary updated

BibTeX
@article{popovic2015problems,
  author = {Dusan Popovic and Alejandro Sifrim and Jesse Davis and Yves Moreau and Bart De Moor},
  title = {{Problems with the nested granularity of feature domains in bioinformatics: the eXtasy case}},
  journal = {BMC Bioinformatics},
  year = {2015},
  doi = {10.1186/1471-2105-16-s4-s2},
}
Figure