ALPaCA: Adapting Llama for Pathology Context Analysis to enable slide-level question answering

Zeyu Gao, Kai He, Weiheng Su, Inês Machado, William McGough, Mercedes Jimenez‐Liñan, Brian Rous, Chunbao Wang, Chengzu Li, Xiaobo Pang, Tieliang Gong, Ming Y. Lu, Faisal Mahmood, Mengling Feng, Mireia Crispin‐OrtuzarmedRxiv 2025 · DOI

TL;DR

This paper introduces ALPaCA, a novel Large Vision-Language Model (LVLM) designed for comprehensive slide-level question answering in digital pathology. By adapting the Llama 3.1 LLM with specialized adaptors and training on a massive dataset of whole-slide images (WSIs), ALPaCA achieves state-of-the-art performance, paving the way for more sophisticated AI-driven diagnostic tools.

What Problem Does This Paper Address?

Digital pathology relies on analyzing whole-slide images (WSIs), which are gigapixel-scale and require integrating information from multiple regions and magnifications. Existing LVLMs, while powerful, are typically limited to analyzing small, predefined regions of interest (ROIs). This limitation prevents them from performing comprehensive, slide-level analysis needed for accurate clinical diagnosis. The challenge lies in developing models that can effectively process the immense scale and hierarchical nature of WSIs to answer complex questions about the entire slide.

Methods

The core of ALPaCA is the adaptation of the Llama 3.1 LLM for pathology. This involves:

  • Hierarchical Patch Embeddings: Using the CONCH vision encoder to extract features from patches across different magnifications, creating hierarchical embeddings that capture multi-scale information.
  • Slide-Level Adaptors: Developing six distinct adaptors (four vision-only: Sampling, ABMIL, KMeans, GMM; and two vision-text interactive: QFormer, LongFormer) to aggregate these patch embeddings into WSI tokens. These adaptors are crucial for compressing the vast WSI data into a format manageable by the LLM.
  • Three-Stage Training: A staged training approach:
  • Comprehension: Freezing the LLM and training only the adaptors to predict WSI descriptions.
  • Instruction: End-to-end training of the LLM and adaptors on general QA tasks.
  • Specialisation: Fine-tuning on specific disease datasets for in-depth analysis.
  • ALPACA-Hybrid: The best-performing variant, combining GMM and LongFormer adaptors, leverages GMM for global visual information and LongFormer for question-specific local visual representations.

Datasets

  • General Slide-Caption Dataset: Curated from TCGA (32 cancer studies) and GTEx (40 non-tumor sites), comprising 35,913 WSIs with corresponding descriptions.
  • General Slide-QA Dataset: Derived from the "General Slide-Caption" dataset, containing 341,051 QA pairs (close-ended and open-ended) across tumor and non-tumor cases.
  • Specific Slide-QA Datasets:
    • Specific Slide-QA (Breast): Focuses on molecular and morphological aspects of breast cancer.
  • Specific Slide-QA (Morphology): Covers gastric cancer, clear cell renal carcinoma, and ovarian cancer.

Key Insights and Ideas

  • Bridging the Scale Gap: The development of specialized slide-level adaptors is critical for enabling LLMs to process gigapixel WSIs, overcoming the limitations of ROI-based analysis.
  • Hybrid Adaptors Excel: Combining global (GMM) and local/interactive (LongFormer) adaptors in ALPACA-Hybrid yields superior performance, suggesting that a multi-faceted approach to WSI representation is most effective.
  • Fine-tuning for Specialization: ALPACA's ability to be fine-tuned on disease-specific datasets demonstrates its adaptability and potential for highly specialized pathology tasks.
  • Dataset Curation is Key: The creation of large-scale, high-quality WSI QA datasets is foundational for advancing slide-level LVLM research.

Key Results

  • Close-ended QA: ALPACA variants achieved over 90% accuracy on close-ended questions, with ALPACA-Hybrid reaching 91.76% overall accuracy.
  • Open-ended QA: Evaluated using a GPT-4-based judger and expert pathologists, ALPACA-Hybrid demonstrated strong performance, achieving a correct rate of 67.40% (TC+PC) and outperforming other variants.
  • Disease-Specific QA: Performance on specialized datasets was comparable to or even exceeded general QA performance, highlighting ALPACA's flexibility. For instance, in "Specific Slide-QA (Breast)", open-ended performance achieved a word-level F1-score of 72.66%.
  • Expert Validation: Expert pathologists confirmed ALPACA-Hybrid's capabilities, noting its potential to identify subtle or overlooked pathological details.

Limitations and Future Work

  • Dataset Volume: While large, the dataset volume is still considered relatively limited, especially for rare tumors.
  • Data Source Differences: GTEx data (postmortem) differs from typical diagnostic settings, and TCGA reports can be noisy due to encompassing multiple slides.
  • Spatial Grounding: ALPACA lacks explicit spatial grounding, meaning it cannot pinpoint specific slide regions associated with its answers. This is a key area for future work, potentially by integrating detection and segmentation models.
  • Reasoning Capabilities: The model does not yet leverage reinforcement learning for decision-making, which could further enhance its reasoning and clinical applicability.
  • Future Directions: Improving dataset quality with per-slide descriptions, developing multi-slide integration frameworks, and enhancing spatial grounding and deep reasoning capabilities are suggested.

Summary updated

Abstract

Abstract Large Vision Language Models (LVLMs) have recently revolutionized computational pathology. LVLMs transform pathology image embeddings into tokens recognizable by large language models, facilitating zero-shot image classification, description generation, question answering, and interactive diagnostics. In clinical practice, pathological assessments often require the analysis of entire tissue slides, integrating information from multiple sub-regions and magnification levels. However, existing LVLM frameworks have been restricted to the analysis of small, predefined regions of interest, lacking the ability to analyze pyramidal, gigapixel-scale whole-slide images (WSIs). In this work, we introduce ALPaCA ( A dapting L lama for Pa thology C ontext A nalysis), and train the first general-purpose slide-level LVLM, leveraging 35,913 WSIs with curated descriptions alongside 341,051 question and answer pairs encompassing diverse diagnoses, procedures, and tissue types. By developing LongFormer, a vision-text interactive slide-level adaptor, and integrating it with a Gaussian mixture model-based prototyping adaptor, followed by training with Llama3.1, ALPaCA achieves superior performance in slide-level question answering, achieving over 90% accuracy in close-ended tests and high accuracy in open-ended questions as evaluated by expert pathologists, highlighting its potential for slide-level computer-aided diagnosis systems. Additionally, we show that ALPaCA can be readily fine-tuned on in-depth, organ-specific, or disease-specific datasets, underscoring its adaptability and utility for specialized pathology tasks.
BibTeX
@misc{gao2025alpaca,
  author = {Zeyu Gao and Kai He and Weiheng Su and Inês Machado and William McGough and Mercedes Jimenez‐Liñan and Brian Rous and Chunbao Wang and Chengzu Li and Xiaobo Pang and Tieliang Gong and Ming Y. Lu and Faisal Mahmood and Mengling Feng and Chen Li and Mireia Crispin‐Ortuzar},
  title = {{ALPaCA: Adapting Llama for Pathology Context Analysis to enable slide-level question answering}},
  year = {2025},
  doi = {10.1101/2025.04.22.25326190},
  howpublished = {medRxiv},
}
Figure