Abstract
In certain types of cancerous tissue, mitotic count has been shown to be associated with tumor proliferation, poor prognosis, and therapeutic resistance. Due to the high inter-rater variability of mitotic counting by pathologists, convolutional neural networks (CNNs) have been employed to reduce the subjectivity of mitosis detection in hematoxylin and eosin (H&E)-stained whole slide images. However, most existing models have performance that lags behind expert panel review and only incorporate visual information. In this work, we demonstrate that pre-trained large-scale vision-language models that leverage both visual features and natural language improve mitosis detection accuracy. We formulate the mitosis detection task as an image captioning task and a visual question answering (VQA) task by including metadata such as tumor and scanner types as context. The effectiveness of our pipeline is demonstrated via comparison with various baseline models using 9,501 mitotic figures and 11,051 hard negatives from the publicly available Mitosis Domain Generalization Challenge (MIDOG22) dataset.
Introduction
In oncology, the "mitotic count"—the number of cells actively dividing—is a critical biomarker for determining tumor aggressiveness and predicting patient response to treatment. However, identifying mitotic figures under a microscope is notoriously difficult. Even expert pathologists often disagree on whether a specific cell is undergoing mitosis or is a "hard negative"—a cell that looks similar but is not actually dividing.
While deep learning models, specifically Convolutional Neural Networks (CNNs), have been used to automate this process, they typically rely solely on raw pixel data. In contrast, human pathologists utilize contextual information, such as tissue type, species, and the technical characteristics of the digital scanner. This research addresses the gap between pixel-only AI and context-aware human reasoning by integrating natural language metadata into the detection process using Large Vision-Language Models (VLMs).
Methodology
The researchers moved away from standard image classification, instead framing mitosis detection using two popular VLM architectures: CLIP and BLIP (Bootstrapping Language-Image Pre-training).
- Image Captioning: The model generates a text description of a tissue tile (e.g., "mitotic, breast cancer, human, aperio").
- Visual Question Answering (VQA): The model is presented with an image and a text prompt containing metadata: "This is an image of [species] [tumor type] taken using scanner [scanner]. Is there mitosis in the image?" The model then predicts a "yes" or "no" answer.
The core innovation is the use of BLIP. Unlike models that only use contrastive learning, BLIP is trained to understand, explain, and answer questions about images. The researchers fine-tuned these models on histopathology tiles, comparing them against traditional vision-only baselines like ResNet-50 and Vision Transformers (ViT).
Data and Preprocessing
The study utilized the MIDOG22 (Mitosis Domain Generalization Challenge) dataset, which includes 354 cases covering various tumor types, species (human, dog, cat), and different digital scanners. The dataset contains 9,501 annotated mitotic figures and 11,051 "hard negatives."
To prepare the data, 50x50 pixel annotations were expanded into 224x224 tiles. Random shifting was used as a form of data augmentation to ensure the models were not biased toward centered features.
Results and Discussion
The BLIP VQA model emerged as the superior approach, outperforming all other models:
- F1 Score: BLIP VQA achieved 0.860, significantly higher than the best vision-only baseline (ImageNet-pre-trained ResNet-50 at 0.832).
- AUC (Area Under Curve): BLIP VQA reached 0.853, compared to 0.831 for the best ResNet baseline.
Key insights from the results include:
- Metadata as Prior Knowledge: Adding metadata (tumor type, scanner) directly into the text prompt improved performance. This suggests the model uses text as a "hint" to adjust its visual expectations, mimicking a pathologist's workflow.
- BLIP vs. CLIP: While CLIP is an industry standard, it failed to outperform vision-only baselines. BLIP’s success is attributed to its more complex training objectives, allowing for a deeper understanding of the relationship between text and pixels.
- Multimodal Synergy: The study demonstrates that language provides a multimodal boost, helping the AI distinguish subtle cellular features that pixels alone might miss.
Conclusion
While promising, the current approach has limitations. The model currently focuses on binary classification (mitosis vs. no mitosis); future work could involve identifying specific mitotic phases or abnormalities. Additionally, to be clinically useful, the model must be adapted for Whole Slide Images (WSIs) using a sliding-window approach to calculate overall mitotic density. Finally, more sophisticated prompt engineering or alternative VLM architectures may further enhance accuracy.
Summary updated