Pathologyvlm: a large vision-language model for pathology image understanding

Dawei Dai, Yuanhui Zhang, Qianlan Yang, Long Xu, Xiaojing Shen, Shuyin Xia, Guoyin WangArtif. Intell. Rev 2025 · DOI

Introduction

General-purpose large vision-language models (VLMs) often struggle with the specialized domain of pathology. While models like GPT-4o excel at identifying everyday objects, they frequently fail to capture the subtle cellular patterns and tissue architectures necessary for medical diagnosis. PathologyVLM addresses this gap by integrating a domain-specific visual encoder with a scale-invariant architecture and a high-quality, curated dataset.

Overcoming the Domain Gap

The authors identify three primary challenges in current medical VLMs:

  1. Domain Gap: Standard visual encoders (e.g., CLIP) are not trained on histopathology slides, leading to suboptimal feature extraction.
  2. Information Loss: The massive scale of pathology slides is typically lost when images are resized to standard 224x224 squares, destroying microscopic diagnostic details.
  3. Data Quality: Public medical datasets often contain "noise," such as non-human samples or non-pathological images, which can degrade model performance.

Methodology

PathologyVLM utilizes a three-stage learning pipeline to ensure diagnostic precision:

  • Specialized Visual Encoder (PLIP): The model employs Pathology Language-Image Pretraining (PLIP), trained using Image-Text Contrastive (ITC) and Image-Text Matching (ITM) losses to align visual features with medical terminology.
  • Scale-Invariant Connector: A dynamic high-resolution strategy processes images in patches, preserving fine-grained cellular details that would otherwise be lost during resizing.
  • Two-Stage Training: The model first undergoes domain alignment through large-scale captioning, followed by instruction tuning on specific Q&A pairs to handle complex diagnostic reasoning.
  • Backbone Architecture: The system leverages Llama3, fine-tuned via Low-Rank Adaptation (LoRA) for computational efficiency.

The Impact of Data Quality

A key finding of the research is that "better data" outperforms "more data." The researchers used LLMs (GLM-4V and Qwen2) to filter public datasets, creating human-specific subsets like PCaption-0.8M. Removing non-human tissue (e.g., mouse or rabbit) and non-pathological images significantly improved downstream performance.

Results and Evaluation

PathologyVLM demonstrates state-of-the-art performance across multiple benchmarks, notably outperforming general-purpose models in zero-shot scenarios.

Examples of PathologyVLM performing captioning, open-set VQA, and closed-set VQA tasks.
  • Supervised VQA: On the PathVQA dataset, the model achieved 92.51% accuracy on closed-set questions, surpassing LLaVA-Med (91.09%) and Quilt-LLaVA (58.68%).
  • Zero-Shot Generalization: In tests on unseen datasets (BACH, OSCC, and ColonPath), PathologyVLM reached an overall accuracy of 62.98%, outperforming GPT-4o (58.00%).
  • Expert Comparison: In a manual evaluation, the model achieved 92.51% accuracy, while human pathologists—provided only with images and no clinical history—achieved 71.15%. This underscores the model's superior pattern recognition capabilities.

Conclusion

PathologyVLM proves that domain-specific specialization is essential for high-stakes medical AI. By combining specialized encoders with high-resolution processing and clean data, the model sets a new standard for pathology image understanding. Future work will focus on incorporating multi-modal clinical context, such as patient history and laboratory results, to further align the model with real-world clinical practice.

Summary updated

BibTeX
@article{dai2025pathologyvlm,
  author = {Dawei Dai and Yuanhui Zhang and Qianlan Yang and Long Xu and Xiaojing Shen and Shuyin Xia and Guoyin Wang},
  title = {{Pathologyvlm: a large vision-language model for pathology image understanding}},
  journal = {Artif. Intell. Rev.},
  year = {2025},
  doi = {10.1007/s10462-025-11190-1},
}
Figure