PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding

Dawei Dai, Yuanhui Zhang, Long Xu, Qianlan Yang, Xiaojing Shen, Shuyin Xia, Guoyin Wang

Pathology serves as the "gold standard" for disease diagnosis, requiring clinicians to perform microscopic analysis of tissue samples to identify subtle cellular abnormalities. While general-purpose Large Language-Vision Models (LLVMs) have demonstrated impressive capabilities in everyday visual tasks, they often falter in specialized medical domains. PA-LLaVA (Pathology-LLaVA) is a novel framework designed to bridge this gap by specializing in human pathology through architectural innovation and rigorous data curation.

Bridging the Gap in Pathology AI

Current general-purpose LLVMs face three primary hurdles when applied to pathology:

  1. Domain Mismatch: Models trained on natural images (e.g., CLIP) lack the sensitivity required to distinguish complex histological patterns.
  2. Information Loss: Standard models typically resize images to small, fixed resolutions (e.g., 224x224), which can destroy the fine-grained cellular details essential for diagnosis.
  3. Data Noise: Existing medical datasets are often cluttered with non-human data (e.g., laboratory mice) or non-pathological images, leading to clinical hallucinations.

The PA-LLaVA Framework

To address these challenges, the researchers developed a specialized three-stage pipeline that optimizes the model for the unique requirements of histological analysis.

Overview of the PA-LLaVA pipeline, including data cleaning, model architecture, and training stages.

Specialized Visual Encoding

Instead of a general CLIP encoder, PA-LLaVA utilizes a Pathology Language-Image Pretraining (PLIP) model. This ensures the model's "vision" is specifically tuned to features like cell nuclei, stroma, and tissue architecture.

Scale-Invariant Connector

To solve the resolution problem, the framework implements a dynamic high-resolution strategy. Rather than simply shrinking a large pathology slide, the connector uses a combination of splitting and resizing. This approach retains original image details, preventing the loss of critical diagnostic features that might otherwise be blurred by standard resizing techniques.

Training Strategy

The model undergoes a three-stage training process:

  • Stage 1 (Pretraining): Utilizing ~827k image-text pairs to establish foundational pathology knowledge.
  • Stage 2 (Domain Alignment): Aligning the LLM (Llama3) with pathology-specific visual features using ~518k pairs.
  • Stage 3 (Instruction Tuning): Fine-tuning on ~35k VQA pairs using LoRA (Low-Rank Adaptation) to enable complex clinical reasoning.

Data Curation: PCaption-0.5M

A significant contribution of this work is the creation of a "clean" human pathology corpus. The authors pulled approximately 1.4 million pairs from sources like Quilt-1M and PubMedVision. They then employed advanced vision models (GLM-4V) and language models (Qwen2) to filter out non-pathological images and non-human data (such as animal samples). This resulted in the PCaption-0.5M dataset, which significantly reduces noise and improves the model's clinical reliability.

Experimental Results

PA-LLaVA demonstrates state-of-the-art performance across multiple benchmarks, proving the efficacy of domain-specific tuning.

Qualitative examples of PA-LLaVA performing caption generation and visual question answering.
  • Supervised VQA: On the PathVQA dataset, PA-LLaVA achieved 92.51% accuracy on closed-set questions, outperforming previous leaders like LLaVA-Med (91.09%).
  • Zero-Shot Performance: In a head-to-head comparison against GPT-4o on the ColonPath dataset, PA-LLaVA achieved 79.96% accuracy, significantly higher than GPT-4o’s 56.12%.
  • Robustness: The model showed a substantial lead in the PMC-VQA (HT-Clean) benchmark, scoring 40.85% compared to LLaVA-Med’s 23.78%, validating the effectiveness of its domain alignment.

Discussion and Future Outlook

The results highlight that for specialized fields like pathology, resolution and data purity are more critical than raw model size. The scale-invariant connector and the removal of animal-based data were key drivers of PA-LLaVA's success.

However, limitations remain. The training data is naturally skewed toward common cancers, and like most LLVMs, the model's internal reasoning remains a "black box." Future work will focus on incorporating whole-slide images (WSIs) and improving dataset balance across rarer organ systems to further enhance clinical utility.

Summary updated

Abstract

The previous advancements in pathology image understanding primarily involved developing models tailored to specific tasks. Recent studies has demonstrated that the large vision-language model can enhance the performance of various downstream tasks in medical image understanding. In this study, we developed a domain-specific large language-vision assistant (PA-LLaVA) for pathology image understanding. Specifically, (1) we first construct a human pathology image-text dataset by cleaning the public medical image-text data for domain-specific alignment; (2) Using the proposed image-text data, we first train a pathology language-image pretraining (PLIP) model as the specialized visual encoder for pathology image, and then we developed scale-invariant connector to avoid the information loss caused by image scaling; (3) We adopt two-stage learning to train PA-LLaVA, first stage for domain alignment, and second stage for end to end visual question \& answering (VQA) task. In experiments, we evaluate our PA-LLaVA on both supervised and zero-shot VQA datasets, our model achieved the best overall performance among multimodal models of similar scale. The ablation experiments also confirmed the effectiveness of our design. We posit that our PA-LLaVA model and the datasets presented in this work can promote research in field of computational pathology. All codes are available at: https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA}{https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA
BibTeX
@inproceedings{dai2024pa,
  author = {Dawei Dai and Yuanhui Zhang and Long Xu and Qianlan Yang and Xiaojing Shen and Shuyin Xia and Guoyin Wang},
  title = {{PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10821785},
}
Figure