Local Attention Graph-Based Transformer for Multi-target Genetic Alteration Prediction

Daniel Reisenbüchler, Sophia J. Wagner, Melanie Boxberg, Tingying Peng

TL;DR

LA-MIL (Local Attention Multiple Instance Learning) is a new Transformer-based framework that predicts multiple genetic mutations directly from massive "gigapixel" pathology slides. By restricting the model's attention to local spatial neighborhoods rather than the entire image at once, the researchers achieved state-of-the-art accuracy—including an 8% boost in predicting microsatellite instability—while significantly reducing the computational burden.

What Problem Does This Paper Address?

In computational pathology, Whole Slide Images (WSIs) are enormous, often containing billions of pixels. To analyze them, researchers typically use Multiple Instance Learning (MIL), where an image is broken into thousands of small "tiles."

Traditional MIL methods often assume these tiles are independent, ignoring the spatial context (how cells relate to their neighbors). While Transformers can model these relationships using "global self-attention," they suffer from quadratic complexity: as the number of tiles increases, the memory and compute power required grows exponentially. This makes global Transformers difficult to scale for the largest slides. The authors ask: Is global context actually necessary, or is local context enough to predict a patient's genetic profile?

Methods

The authors introduce LA-MIL, a pipeline that combines Graph Neural Networks with Transformers:

  1. Preprocessing & Feature Extraction: WSIs are tessellated into tiles. A pretrained CNN (KimiaNet/DenseNet-121) converts each tile into a compact feature vector.
  2. Spatial Graph Construction: Instead of letting every tile "talk" to every other tile, the model builds a k-Nearest Neighbor (kNN) graph based on the tiles' actual coordinates on the slide.
  3. Local Attention Transformer: The model uses "Local Attention" layers. For any given tile, the Transformer only calculates attention scores for its $k$ nearest neighbors. This introduces a "spatial inductive bias," forcing the model to focus on local morphological structures.
  4. Multi-target Prediction: Unlike many models that train a separate network for every gene, LA-MIL uses a single backbone to predict multiple biomarkers (e.g., MSI, TMB, and various mutations) simultaneously. They use a weighted Binary Cross-Entropy loss to handle the fact that some mutations are much rarer than others.

Datasets

The model was evaluated on two major cohorts from The Cancer Genome Atlas (TCGA):

  • TCGA-CRC: 594 slides of colorectal cancer.
  • TCGA-STAD: 440 slides of stomach adenocarcinoma.

The targets included clinical biomarkers like Microsatellite Instability (MSI) and Tumor Mutational Burden (TMB), as well as mutations in specific genes like BRAF, KRAS, PIK3CA, and TP53.

Key Insights and Ideas

  • Local is "Enough": The most significant finding is that local self-attention performs on par with (or better than) global modules. This suggests that the biological signals for genetic alterations are often contained within local tissue architecture rather than long-range dependencies across the entire slide.
  • Efficiency via Sparsity: By using kNN graphs, the model bypasses the $O(N^2)$ complexity of standard Transformers. This makes it a "computationally light" alternative that can handle varying slide shapes and sizes without the "holes" or artifacts created by rigid window-based attention.
  • Synergy in Multi-tasking: Training one model to predict multiple targets allowed the network to learn shared representations that improved overall performance, particularly for MSI prediction.

Key Results

  • State-of-the-Art in CRC: On the TCGA-CRC dataset, LA-MIL achieved an AUROC of 0.85 for MSI prediction, outperforming the previous benchmark of 0.77 (an 8% improvement).
  • Competitive Performance: Across almost all genetic mutations (BRAF, KRAS, etc.), LA-MIL either outperformed or matched existing instance-wise and global Transformer methods.
  • Interpretability: By visualizing the local attention scores, the authors showed that the model naturally focuses on high-importance regions like hyperchromatic nuclear chromatin and crowded glandular structures, providing a "heat map" for pathologists to inspect.

Limitations and Future Work

  • Single Slide Constraint: Currently, the model only processes one WSI per patient. However, patients often have multiple slides available. The authors note that future work should investigate how to combine information across multiple WSIs while properly scaling coordinates.
  • Interpretability Gap: While the model highlights "high-attention" regions, the authors admit there are still no clearly defined visual patterns that humans can easily link to specific mutated genes, making quantitative evaluation of the "explanations" difficult.
  • Dataset Diversity: The study focused on gastrointestinal cancers; testing the local attention hypothesis on other cancer types (like lung or breast) remains an open area for research.

Summary updated

Abstract

Classical multiple instance learning (MIL) methods are often based on the identical and independent distributed assumption between instances, hence neglecting the potentially rich contextual information beyond individual entities. On the other hand, Transformers with global self-attention modules have been proposed to model the interdependencies among all instances. However, in this paper we question: Is global relation modeling using self-attention necessary, or can we appropriately restrict self-attention calculations to local regimes in large-scale whole slide images (WSIs)? We propose a general-purpose local attention graph-based Transformer for MIL (LA-MIL), introducing an inductive bias by explicitly contextualizing instances in adaptive local regimes of arbitrary size. Additionally, an efficiently adapted loss function enables our approach to learn expressive WSI embeddings for the joint analysis of multiple biomarkers. We demonstrate that LA-MIL achieves state-of-the-art results in mutation prediction for gastrointestinal cancer, outperforming existing models on important biomarkers such as microsatellite instability for colorectal cancer. Our findings suggest that local self-attention sufficiently models dependencies on par with global modules. Our LA-MIL implementation is available at https://github.com/agentdr1/LA_MIL.
BibTeX
@inproceedings{reisenbuchler2022local,
  author = {Daniel Reisenbüchler and Sophia J. Wagner and Melanie Boxberg and Tingying Peng},
  title = {{Local Attention Graph-Based Transformer for Multi-target Genetic Alteration Prediction}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16434-7\_37},
}

Loading related content...

Figure