Improving Mitosis Detection on Histopathology Images Using Large Vision-Language Models

Ruiwen Ding, James Brian Hall, Neil A. Tenenholtz, Kristen Severson

TL;DR

Researchers have developed a new approach to detecting mitosis (cell division) in cancer biopsies by treating the problem as a conversation between an AI and an image. By using Large Vision-Language Models (VLMs) like BLIP to incorporate clinical metadata—such as the type of tumor and the specific scanner used—the team achieved significantly higher accuracy than traditional "vision-only" AI models.

What Problem Does This Paper Address?

In oncology, the "mitotic count" (how many cells are currently dividing) is a critical biomarker for determining how aggressive a tumor is and how a patient might respond to treatment. However, identifying mitotic figures under a microscope is notoriously difficult. Even expert pathologists often disagree on whether a specific cell is undergoing mitosis or is just a "hard negative"—a cell that looks similar but isn't actually dividing.

While deep learning models (specifically Convolutional Neural Networks or CNNs) have been used to automate this, they typically only "see" the raw pixels. In contrast, human pathologists use context: they know the tissue type, the species, and the technical quirks of the digital scanner. This paper addresses the gap between pixel-only AI and context-aware human reasoning by integrating natural language metadata into the detection process.

Methods

The authors moved away from standard image classification and instead framed mitosis detection using two popular Vision-Language Model (VLM) architectures: CLIP and BLIP.

  1. Image Captioning: The model is asked to generate a text description of a tissue tile (e.g., "mitotic, breast cancer, human, aperio").
  2. Visual Question Answering (VQA): The model is given an image and a text prompt containing metadata: "This is an image of [species] [tumor type] taken using scanner [scanner]. Is there mitosis in the image?" The model then predicts a "yes" or "no" answer.

The core innovation is the use of BLIP (Bootstrapping Language-Image Pre-training). Unlike models that only use contrastive learning (matching images to labels), BLIP is trained to understand, explain, and answer questions about images. The researchers fine-tuned these models on histopathology tiles, comparing them against traditional vision-only baselines like ResNet-50 and Vision Transformers (ViT).

Datasets

The study utilized the MIDOG22 (Mitosis Domain Generalization Challenge) dataset.

  • Scope: 354 cases covering various tumor types, species (human, dog, cat), and different digital scanners.
  • Content: 9,501 annotated mitotic figures and 11,051 "hard negatives" (non-mitotic cells that are easily confused with mitotic ones).
  • Preprocessing: The team expanded 50x50 pixel annotations into 224x224 tiles and used random shifting as a form of data augmentation to ensure the models weren't just looking at the center of the frame.

Key Insights and Ideas

  • Metadata as "Prior Knowledge": The most significant "aha" moment was that adding metadata (tumor type, scanner) directly into the text prompt improved performance. This suggests the model uses the text as a "hint" to adjust its visual expectations, much like a pathologist does.
  • BLIP vs. CLIP: While CLIP is a industry standard, it actually failed to outperform vision-only baselines in this study. The authors attribute BLIP’s success to its more complex training objectives (like language modeling and matching), which allow for a deeper "understanding" of the relationship between text and pixels.
  • Multimodal Synergy: The paper proves that language isn't just an extra label; when properly integrated, it provides a "multimodal" boost that helps the AI distinguish between subtle cellular features that pixels alone might miss.

Key Results

The BLIP VQA model was the clear winner, outperforming all other models:

  • F1 Score: BLIP VQA achieved 0.860, significantly higher than the best vision-only baseline (ImageNet-pre-trained ResNet-50 at 0.832).
  • AUC (Area Under Curve): BLIP VQA reached 0.853, compared to 0.831 for the best ResNet baseline.
  • Zero-shot potential: Even without specific fine-tuning on the mitosis labels, BLIP models showed a basic ability to recognize these features, though fine-tuning was necessary to reach peak performance.
  • Metadata Impact: Including metadata in the VQA prompt increased the F1 score from 0.695 (no metadata) to 0.704 in a zero-shot setting, confirming that context matters.

Limitations and Future Work

The authors acknowledge a few key areas for growth:

  • Binary Focus: Currently, the model only performs binary classification (mitosis vs. no mitosis). Future work could involve identifying specific phases of mitosis (prophase, metaphase, etc.) or identifying abnormalities.
  • Slide-Level Analysis: This study focused on small "tiles." To be useful in a clinic, the model needs to be applied to "Whole Slide Images" (WSIs) using a sliding-window approach to calculate overall mitotic density.
  • Prompt Engineering: While they used a standard VQA prompt, more sophisticated "prompt engineering" or different VLM architectures might unlock even higher accuracy.

Summary updated

Abstract

In certain types of cancerous tissue, mitotic count has been shown to be associated with tumor proliferation, poor prognosis, and therapeutic resistance. Due to the high inter-rater variability of mitotic counting by pathologists, convolutional neural networks (CNNs) have been employed to reduce the subjectivity of mitosis detection in hematoxylin and eosin (H&E)-stained whole slide images. However, most existing models have performance that lags behind expert panel review and only incorporate visual information. In this work, we demonstrate that pre-trained large-scale vision-language models that leverage both visual features and natural language improve mitosis detection accuracy. We formulate the mitosis detection task as an image captioning task and a visual question answering (VQA) task by including metadata such as tumor and scanner types as context. The effectiveness of our pipeline is demonstrated via comparison with various baseline models using 9,501 mitotic figures and 11,051 hard negatives (non-mitotic figures that are difficult to characterize) from the publicly available Mitosis Domain Generalization Challenge (MIDOG22) dataset.
BibTeX
@inproceedings{ding2024improving,
  author = {Ruiwen Ding and James Brian Hall and Neil A. Tenenholtz and Kristen Severson},
  title = {{Improving Mitosis Detection on Histopathology Images Using Large Vision-Language Models}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=RvAsMUEqty},
}

Loading related content...

Figure