Abstract
Pathology faces a significant "data desert" compared to general vision-language tasks, primarily due to the scarcity of high-quality, granular image-text pairs. We introduce PathGen-1.6M, a massive dataset of 1.6 million pathology image-text pairs generated via a novel multi-agent collaboration pipeline. By leveraging Whole Slide Images (WSIs) and clinical reports, our system extracts representative patches and generates expert-level captions. Models trained on this data, PathGen-CLIP and PathGen-LLaVA, establish new state-of-the-art benchmarks in zero-shot classification and multimodal reasoning, even surpassing general-purpose models like GPT-4V in specialized pathological tasks.
Addressing the Pathology "Data Desert"
In general computer vision, models like CLIP benefit from billions of web-scraped image-text pairs. However, pathology datasets are often limited in scale, low in resolution, or paired with noisy, non-clinical text. While repositories like The Cancer Genome Atlas (TCGA) provide high-resolution WSIs, they typically offer only slide-level labels, lacking the patch-level descriptions necessary for training robust Vision-Language Models (VLMs). PathGen-1.6M bridges this gap by automating the generation of high-quality, granular captions for microscopic imagery at an unprecedented scale.
Multi-Agent Collaboration Pipeline
The core innovation of this work is a multi-agent system designed to transform unlabeled WSIs into a supervised dataset. This pipeline consists of four key stages:
- Representative Patch Extraction: A "Prompt Agent" (GPT-4) identifies critical features from pathology reports. A specialized CLIP model then retrieves relevant patches, followed by K-means clustering to ensure morphological diversity.
- Description Generation: A custom Large Multimodal Model (LMM), PathGen-LLaVA-desp, generates detailed visual descriptions focusing on clinical features like nuclear atypia and cellular arrangement.
- Description Revision: To mitigate hallucinations, a dedicated "Revise LMM Agent" identifies and corrects inaccuracies in the generated text.
- Description Summarization: A Llama-2 based agent condenses these descriptions into concise captions suitable for CLIP's token limits.
The PathGen-1.6M Dataset
The resulting dataset, PathGen-1.6M, comprises 1.6 million image-caption pairs derived from approximately 7,300 WSIs across 27 tissue types in the TCGA repository. Additionally, the authors curated 200K pathology-specific instruction-following samples for fine-tuning conversational agents. Evaluation was conducted across nine zero-shot classification datasets (including Pcam, CRC100K, and BACH) and the PathMMU benchmark for multimodal reasoning.
Experimental Results
The effectiveness of PathGen-1.6M was evaluated across various benchmarks, including zero-shot classification, WSI-level analysis, and multimodal reasoning.
Zero-Shot Classification
PathGen-CLIP models demonstrate significant improvements over previous state-of-the-art (SOTA) pathology models. On average, PathGen-CLIP outperformed competitors like PathCLIP and QuiltNet by 8.1% across nine datasets. Notably, on the Pcam dataset, it achieved a 30.5% improvement over QuiltNet.
Multimodal Reasoning and WSI Analysis
PathGen-LLaVA achieved a score of 58.4% on the PathMMU benchmark, outperforming GPT-4V (49.8%) and the previous best pathology LMM, Quilt-LLaVA (41.5%). Furthermore, in WSI-level classification using Multiple Instance Learning, PathGen-CLIP reached an average AUC of 92.6%, proving its utility as a powerful feature encoder for clinical applications.
Few-Shot Efficiency
The model exhibits remarkable data efficiency. With just 8 labeled samples per class, PathGen-CLIP reached nearly 94% accuracy in lung cancer classification, highlighting its potential for deployment in scenarios where expert annotations are scarce.
Conclusion and Future Directions
PathGen-1.6M demonstrates that high-quality synthetic data, refined through multi-agent revision, can surpass noisy web-scraped data for training specialized medical AI. While the current pipeline relies on existing pathology reports for initial guidance, future work will explore self-guided discovery of interesting regions within unlabelled slides. The success of PathGen-CLIP-L also suggests that pathology models can continue to benefit from the scaling laws observed in general AI.
Summary updated