Statistically valid explainable black-box machine learning: applications in sex classification across species using brain imaging

Tingshan Liu, Jayanta Dey, B. Xu, Samuel Alldritt, Karl‐Heinz Nenning, Kyoungseob Byeon, Ting Xu, Joshua T VogelsteinbioRxiv 2025 · DOI

Abstract

Sex classification using neuroimaging data has the potential to revolutionize personalized diagnostics by revealing subtle structural brain differences that underlie sex-specific disease risks. Despite the promise of machine learning, traditional methods often fall short in providing both high classification accuracy and interpretable, statistically validated feature importance scores for high-dimensional imaging data. This gap is particularly evident when conventional techniques such as random forests, LIME, and SHAP are applied, as they struggle with complex feature interactions and managing noise in large datasets. We address this challenge by developing an integrated framework that combines Oblique Random Forests (ORFs) with a novel, permutation-based feature importance testing algorithm. ORFs extend traditional random forests by employing oblique decision boundaries through linear combinations of features, thereby capturing intricate interactions inherent in neuroimaging data. Our feature importance testing method, NEOFIT, rigorously quantifies the significance of each feature by generating null distributions and corrected p-values. We first validate our approach using simulated datasets, establishing its robustness and scalability under controlled conditions. We then apply our method to classify sex from both voxel-wise structural MRI and cortical thickness data in humans and macaques, facilitating direct cross-species comparisons. ORFs achieves AUC >0.80 on human data, and >0.70 on macaque data, while NEOFIT identifies statistically significant features aligned with sex-dimorphic neuroanatomy. Our results demonstrate that the proposed framework not only enhances classification performance but also provides clear, interpretable insights into the neuroanatomical features that distinguish sexes. These methodological advancements pave the way for improved diagnostic tools and contribute to a deeper understanding of the evolutionary basis of sex differences in brain structure.

Introduction

In neuroimaging, researchers often face a trade-off between predictive power and interpretability. While modern machine learning models can predict biological sex with high accuracy, they typically operate as "black boxes." Traditional explanation tools like LIME or SHAP often struggle with the extreme dimensionality of brain scans—which can involve millions of voxels—and the complex, non-linear interactions between disparate brain regions. Furthermore, most existing feature importance scores lack statistical rigor, identifying "important" features without determining if that importance is statistically significant or merely a product of noise. This study introduces a framework designed to bridge this interpretability gap by providing high-performance classification alongside rigorous, statistically valid insights into the underlying neuroanatomy.

Methodology

The proposed framework integrates two primary components designed for high-dimensional biological data:

  1. Oblique Random Forests (ORFs): Standard Random Forests split data based on a single feature at a time (axis-aligned). In contrast, ORFs utilize linear combinations of multiple features at each split, allowing for "oblique" decision boundaries. This study specifically employs SPORF (Sparse Projection ORF) to handle irrelevant features and MORF (Manifold ORF) to account for the spatial correlation inherent in voxel data.
  2. NEOFIT (Neuro-Explainable Optimal Feature Importance Testing): This novel permutation-based testing framework generates null distributions for feature importance. By shuffling trees within an ensemble rather than retraining entire forests, NEOFIT significantly reduces computational overhead. It also incorporates Bonferroni-Holm corrections to address the multiple comparisons problem common in voxel-wise analysis.

Experimental Validation

The framework was validated across simulated benchmarks and large-scale biological datasets. Simulation tests included the MNIST handwritten digits and the Trunk dataset, a synthetic high-dimensional benchmark designed to test feature selection efficiency.

Biological validation utilized human MRI data from over 14,000 subjects (aggregated from sources like FCON1000 and the Healthy Brain Network) and macaque MRI data from 592 rhesus macaques via the PRIME-DE consortium. These datasets included both voxel-wise structural MRI and parcellated cortical thickness (CTh).

Simulation Results

In benchmark tests, NEOFIT consistently outperformed standard Random Forests, LIME, and SHAP. In the Trunk simulation, NEOFIT achieved over 80% accuracy using a significantly smaller subset of features than competing methods, demonstrating its efficiency in identifying the most informative variables in high-dimensional spaces.

NEOFIT performance on simulation benchmarks (Trunk and MNIST) compared to LIME, RF, and SHAP, demonstrating superior feature selection efficiency.

Results and Discussion

Sex Classification and Hyperparameter Sensitivity

The framework achieved an Area Under the Curve (AUC) of approximately 0.98 for human MRI volume data and 0.81 for cortical thickness. Macaque classification was robust, with AUCs ranging from 0.70 to 0.73. Analysis of the models revealed that performance was sensitive to hyperparameters, particularly the number of feature combinations and the maximum features considered at each split.

Hyperparameter tuning for human and macaque sex classification, showing the impact of feature combinations on model accuracy across different parcellation schemes.

Evolutionary Conservation and Divergence

By applying NEOFIT, the researchers identified both shared and species-specific patterns of sex dimorphism. Features in the limbic system, including the amygdala and hippocampus, appeared conserved across both humans and macaques. However, divergence was noted in higher-order regions: humans exhibited more specialized sex differences in cognitive areas, whereas macaques showed more prominent differences in sensory and motor-associated regions.

Conclusion

This research demonstrates that high classification accuracy in neuroimaging does not require a sacrifice in interpretability. By combining models that capture feature interactions (ORFs) with a statistically rigorous testing suite (NEOFIT), the framework provides a valid pathway for "opening the black box" of machine learning in medicine. Future work will focus on reducing the computational demands of voxel-wise permutations and exploring transfer learning to adapt large-scale human models to smaller non-human primate datasets.

Summary updated

Abstract

Sex classification using neuroimaging data has the potential to revolutionize personalized diagnostics by revealing subtle structural brain differences that underlie sex-specific disease risks. Despite the promise of machine learning, traditional methods often fall short in providing both high classification accuracy and interpretable, statistically validated feature importance scores for high-dimensional imaging data. This gap is particularly evident when conventional techniques such as random forests, LIME, and SHAP are applied, as they struggle with complex feature interactions and managing noise in large datasets. We address this challenge by developing an integrated framework that combines Oblique Random Forests (ORFs) with a novel, permutation-based feature importance testing algorithm. ORFs extend traditional random forests by employing oblique decision boundaries through linear combinations of features, thereby capturing intricate interactions inherent in neuroimaging data. Our feature importance testing method, NEOFIT, rigorously quantifies the significance of each feature by generating null distributions and corrected p-values. We first validate our approach using simulated datasets, establishing its robustness and scalability under controlled conditions. We then apply our method to classify sex from both voxel-wise structural MRI and cortical thickness data in humans and macaques, facilitating direct cross-species comparisons. ORFs achieves AUC >0.80 on human data, and >0.70 on macaque data, while NEOFIT identifies statistically significant features aligned with sex-dimorphic neuroanatomy. Our results demonstrate that the proposed framework not only enhances classification performance but also provides clear, interpretable insights into the neuroanatomical features that distinguish sexes. These methodological advancements pave the way for improved diagnostic tools and contribute to a deeper understanding of the evolutionary basis of sex differences in brain structure.
BibTeX
@misc{liu2025statistically,
  author = {Tingshan Liu and Jayanta Dey and B. Xu and Samuel Alldritt and Karl‐Heinz Nenning and Kyoungseob Byeon and Ting Xu and Joshua T Vogelstein and Joshua T. Vogelstein},
  title = {{Statistically valid explainable black-box machine learning: applications in sex classification across species using brain imaging}},
  year = {2025},
  doi = {10.1101/2025.07.09.664026},
  howpublished = {bioRxiv},
}
Figure