Co-Attentive Cross-Modal Deep Learning for Medical Evidence Synthesis and Decision Making

Devin Taylor, Simeon E. Spasov, Pietro LiòarXiv 2019 · arXiv

Abstract

Complex diseases like Parkinson’s Disease (PD) are driven by intricate interactions across biological scales, from epigenetic markers to structural brain changes. Traditional deep learning approaches often fail to capture these cross-modal dependencies effectively or provide biological interpretability. This paper introduces a Multi-Head Co-Attention (MHCA) architecture that learns a joint similarity distribution between 3D SPECT brain scans and DNA methylation data. Our model achieves state-of-the-art accuracy (98.35%) with 53% fewer parameters than standard late-fusion models. Furthermore, the co-attention mechanism enables the discovery of novel biological links, specifically highlighting the interferon-gamma pathway's role in PD via DNA methylation.

Introduction

Diagnosing and understanding complex pathologies like Parkinson’s Disease (PD) requires synthesizing data from diverse sources, such as 3D medical imaging and high-dimensional genomic sequences. Standard "late fusion" approaches process these modalities independently and concatenate features at the final layer. This often results in inefficient learning and a "black box" model that lacks clinical interpretability. To address these challenges, we propose a co-attentive framework that actively weights features from different modalities based on their mutual context.

Methods: Multi-Head Co-Attention (MHCA)

The core of our approach is the Multi-Head Co-Attention (MHCA) mechanism, inspired by Transformer architectures in natural language processing. Unlike self-attention, co-attention computes a similarity matrix between two distinct data streams: 3D spatial features from SPECT images and 1D epigenetic features.

The MHCA architecture integrates image and epigenetic features through bidirectional weighting and residual connections.

The process involves several key stages:

  1. Feature Extraction: A 3D CNN encodes SPECT images to extract spatial features. Simultaneously, epigenetic data (DNA methylation) undergoes targeted feature selection via XGBoost and Recursive Feature Elimination to align its dimensionality with the image latent space.
  2. Bidirectional Weighting: The model calculates two sets of attention coefficients ($\alpha_q$ and $\alpha_v$), determining how much each modality should inform the other. This bidirectional approach reflects the interdependent nature of medical data.
  3. Integration: Weighted features are combined using residual connections and layer normalization before passing to a final linear layer for prediction.

Data and Experimental Setup

We utilized data from the Parkinson’s Progression Markers Initiative (PPMI), including 121 healthy controls and 287 PD patients.

  • SPECT Images: 3D scans modeling dopamine-transporter functionality.
  • DNA Methylation (DNA-m): Epigenetic markers filtered from ~765,000 features down to 441 predictive markers. This upfront selection ensures that attention weights remain directly mappable to specific biological markers.

Results and Discussion

The MHCA model significantly outperforms existing benchmarks. It achieved an accuracy of 98.35% and an AUROC of 0.976, surpassing the previous state-of-the-art 3D-CNN (96.00%) and a baseline concatenation model (97.12%). Notably, the model is highly efficient, utilizing only ~2 million parameters—a 53% reduction compared to the concatenation baseline.

Interpretability and Biological Discovery

A key advantage of MHCA is its inherent interpretability. By visualizing the attention masks, we confirmed the model focuses on the putamen, the region most associated with dopaminergic cell loss in PD.

Qualitative validation showing the model's attention focused on the putamen region in a 3D SPECT scan.

Beyond imaging, the model identified DNA methylation markers linked to the interferon-gamma (IFN-$\gamma$) pathway. While previously linked to PD through gene expression, our findings suggest a novel epigenetic regulatory mechanism for this pathway in PD progression, potentially influenced by environmental factors.

Conclusion

The MHCA framework demonstrates that co-attention is a powerful tool for multimodal medical data integration. By moving beyond simple concatenation, we achieve higher accuracy, greater parameter efficiency, and direct biological interpretability. Future work will focus on scaling the attention mechanism to handle higher-dimensional data natively and exploring how perturbations in one modality affect the attention weights of another.

Summary updated

Abstract

Modern medicine requires generalised approaches to the synthesis and integration of multimodal data, often at different biological scales, that can be applied to a variety of evidence structures, such as complex disease analyses and epidemiological models. However, current methods are either slow and expensive, or ineffective due to the inability to model the complex relationships between data modes which differ in scale and format. We address these issues by proposing a cross-modal deep learning architecture and co-attention mechanism to accurately model the relationships between the different data modes, while further reducing patient diagnosis time. Differentiating Parkinson's Disease (PD) patients from healthy patients forms the basis of the evaluation. The model outperforms the previous state-of-the-art unimodal analysis by 2.35%, while also being 53% more parameter efficient than the industry standard cross-modal model. Furthermore, the evaluation of the attention coefficients allows for qualitative insights to be obtained. Through the coupling with bioinformatics, a novel link between the interferon-gamma-mediated pathway, DNA methylation and PD was identified. We believe that our approach is general and could optimise the process of medical evidence synthesis and decision making in an actionable way.
BibTeX
@misc{taylor2019co,
  author = {Devin Taylor and Simeon E. Spasov and Pietro Liò},
  title = {{Co-Attentive Cross-Modal Deep Learning for Medical Evidence Synthesis and Decision Making}},
  year = {2019},
  eprint = {1909.06442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.06442},
}
Figure