Machine learning models have achieved remarkable success in analyzing medical images. By evaluating standard clinical scans (such as CT, MRI, and PET) or digital pathology slides, these models can predict patient survival, treatment response, and even specific genetic mutations without requiring invasive tissue biopsies.
For example, Qifan Ma demonstrated that CT-based image features can accurately predict how patients with oral squamous cell carcinoma will respond to immunotherapy (Predicting response to immunotherapy in oral squamous cell carcinoma via a CT-based radiomics model). Similarly, Jiangwei Lao et al. built a deep learning model that leverages standard MR images to predict survival outcomes in glioblastoma multiforme patients (A Deep Learning-Based Radiomics Model for Prediction of Survival in Glioblastoma Multiforme).
Yet, beneath these high performance metrics lies a profound paradox: the features driving these predictions are almost entirely detached from human biological understanding. A model might achieve an outstanding area under the curve (AUC) by identifying complex mathematical patterns in the pixels, but clinicians cannot see or explain why those patterns correspond to a specific disease state. This disconnect is known as the biological interpretability gap.
Why Mathematical Textures Are Not Biological Insights
To understand why this gap exists, we must look at what machine learning models actually extract from an image. In traditional "radiomics" or "pathomics" pipelines, algorithms compute thousands of quantitative descriptors from a region of interest. These include shape metrics, intensity histograms, and advanced texture features like Gray-Level Co-occurrence Matrices (GLCM), which measure the spatial relationships between pixels.
As Chintan Parmar et al. demonstrated, these high-dimensional features can be mathematically clustered into stable, non-redundant signatures that strongly correlate with patient survival across different cancer types (Radiomic feature clusters and Prognostic Signatures specific for Lung and Head & Neck cancer). However, a statistical cluster of "high-frequency wavelet textures" does not inherently mean anything to a medical professional. Does this texture represent tumor hypoxia? Is it a sign of dense extracellular matrix? Or is it simply an artifact of scanner calibration?
[ Medical Image (CT/MRI) ] ───> [ Abstract Mathematical Textures ] ───> [ High Accuracy Prediction ]
│
└─── [ Biological Meaning? ] ───> ??? (The Interpretability Gap)
The problem deepens with deep learning models, which bypass handcrafted features entirely to learn latent representations directly from raw pixels. While visualization techniques like saliency maps or attention mechanisms can highlight where a network is looking, they fail to explain what biological properties triggered the prediction. As Zhen Han et al. observed in their work on digital pathology slides, knowing that a model focuses on a specific patch of tissue does not reveal the underlying biological drivers or genomic pathways of a patient's treatment response (Development and interpretation of a pathomics-driven ensemble model for predicting the response to immunotherapy in gastric cancer).
The Danger of Spurious Correlations
Why can we not simply trust the high accuracy of these models and ignore the lack of biological explanation? The danger is that without a mechanistic link, models are highly susceptible to exploiting confounding variables or dataset-specific artifacts.
Consider the domain of cardiac imaging. J. Weston Hughes et al. developed a deep learning framework capable of predicting blood-based biomarkers, such as hemoglobin levels, directly from echocardiogram videos (Deep Learning Prediction of Biomarkers from Echocardiogram Videos). While the model performs exceptionally well, the authors explicitly note that the exact biological mechanisms by which systemic conditions like anemia alter microscopic ultrasound textures or motion remain unverified.
Without establishing a clear physiological cause, it is impossible to guarantee that the model is tracking genuine pathology rather than subtle, systemic biases in how different patient cohorts are scanned.
Grounding Pixels in Pathways via Radiogenomics
To bridge this chasm, a growing direction in biomedical data science seeks to move away from purely predictive models toward frameworks that explicitly map macroscopic imaging phenotypes to microscopic molecular profiles—a field known as radiogenomics.
The core intuition is simple: macroscopic tissue changes visible on a clinical scan are the downstream physical consequences of microscopic genetic and cellular activity. By pairing imaging data with multi-omics data (such as transcriptomics, proteomics, or single-cell sequencing), researchers can decode the biological meaning of abstract mathematical features.
[ Microscopic Genetic Activity ] ───> [ Cellular/Tissue Changes ] ───> [ Macroscopic Imaging Phenotype ]
(e.g., IDH Mutation) (e.g., Low Angiogenesis) (e.g., Decreased Blood Volume)
A clear example of this mechanistic grounding can be seen in a study on human gliomas by Philipp Kickingereder et al. (IDH mutation status is associated with a distinct hypoxia/angiogenesis transcriptome signature which is non-invasively predictable with rCBV imaging in human glioma). Instead of merely training an algorithm to find statistical correlations between MRI pixels and IDH mutation status, the authors traced the complete biological chain of custody:
- The IDH mutation alters a specific hypoxia/angiogenesis transcriptome signature.
- This molecular shift physically suppresses blood vessel growth (angiogenesis) at the cellular level.
- This lack of vascularization manifests as a measurable decrease in regional cerebral blood volume (rCBV) on macroscopic MR images.
By establishing this causal pathway, the imaging biomarker ceases to be a black box; it becomes a logical, interpretable proxy for the tumor's underlying genetic state.
Similarly, Xingping Zhang et al. have leveraged deep learning frameworks to bridge macroscopic CT textures with microscopic genetic expressions and immune markers in non-small cell lung cancer (Exploring non-invasive precision treatment in non-small cell lung cancer patients through deep learning radiomics across imaging features and molecular phenotypes). By forcing computational models to learn representations that correlate simultaneously with visual textures and molecular phenotypes, they ensure that the extracted features are biologically grounded from the start.
The Path Forward: Mechanistic Interpretability
Closing the biological interpretability gap requires a fundamental shift in how the medical imaging community evaluates artificial intelligence. High performance on an isolated test dataset is no longer the sole benchmark for success. True clinical validation demands that computational features correspond to observable pathological traits or known biological pathways.
Whether it is mapping the spatial arrangement of tumor-infiltrating lymphocytes in digital pathology slides (Ruiwen Ding et al., Image analysis reveals molecularly distinct patterns of TILs in NSCLC associated with treatment outcome) or linking macroscopic metabolic heterogeneity to cellular protein activity (Xueyuan Lu et al., Neuroblastoma with high ASPM reveals pronounced heterogeneity and poor prognosis), the objective remains identical. By anchoring abstract mathematical descriptors to the fundamental laws of tumor biology, researchers are transforming black-box algorithms into trusted, mechanistic windows into human disease.
Key Foundational Work
Chapter 17: Bioimage Informatics for Systems Pharmacology (2013) — contributed 2 concept elements to this direction.