Quilt-1M: One Million Image-Text Pairs for Histopathology

Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, Linda Shapiro

TL;DR

Researchers from the University of Washington and the Allen Institute for AI have introduced Quilt-1M, the largest vision-language dataset for histopathology to date, featuring one million image-text pairs. By leveraging over 1,000 hours of educational YouTube videos and using LLMs to clean noisy narrations, they developed QuiltNet, a model that significantly outperforms existing state-of-the-art medical AI in zero-shot classification and cross-modal retrieval.

What Problem Does This Paper Address?

In the world of computer vision, models like CLIP have revolutionized representation learning by training on billions of image-text pairs from the internet. However, histopathology—the study of diseased tissue via microscope—has remained a "data desert."

Standard histopathology datasets are often limited to simple categorical labels (e.g., "cancer" vs. "normal"). This is a massive oversimplification of a field that involves thousands of evolving disease subtypes and complex morphological patterns. While natural language descriptions provide the "dense" signal needed to capture this complexity, curated medical image-text datasets are scarce and small, typically topping out at around 200,000 samples. The authors aim to bridge this gap by finding a new, massive source of expert-level medical knowledge.

Methods

The authors’ core innovation is a sophisticated curation pipeline that turns YouTube into a structured medical dataset.

Figure 1: Overview of QUILT curation pipeline. We identify relevant histopathology YouTube videos in Search. For Image extraction, we find and de-noise histopathology frames using trained models. In Text section, we rely on a conventional Automatic Speech Recognition (ASR) model and leverage Unified Medical Language System (UMLS) and large language models (LLMs) for post- processing and ASR error correction. Relevant sub-pathology, medical and region-of-interest (ROI) text are extracted using an LLM. Finally, domain-specific algorithms are used to Pair images and text, eliminating duplicates to yield QUILT, a richly annotated image-text dataset for histopathology.

1. The YouTube Pipeline

The team identified 1,087 hours of educational histopathology videos. To ensure quality, they filtered for a "narrative style"—videos where experts spend significant time panning and zooming on Whole-Slide Images (WSIs) while describing what they see.

2. Multi-Modal Extraction

  • Image Extraction: They used FFmpeg to identify keyframes and trained an ensemble of classifiers to distinguish actual histopathology slides from "noise" like PowerPoint slides or the presenter's face.
  • Text Denoising: This is the clever part. Automatic Speech Recognition (ASR) often mangles complex medical terms (e.g., transcribing "serous" as "serious"). The authors used a four-step process involving the Unified Medical Language System (UMLS) and GPT-3.5 to cross-check and correct these transcripts within their medical context.

3. Alignment

Finally, they used domain-specific algorithms to pair images with specific snippets of text based on timing and keyword overlap, ensuring that the description actually matches the visual features shown on screen.

Datasets

  • QUILT: The primary new dataset, consisting of 802,144 image-text pairs derived solely from YouTube.
  • QUILT-1M: A combined dataset totaling 1 million pairs, merging QUILT with existing open-source data from Twitter (OpenPath), PubMed, and LAION.
  • Evaluation Suite: The models were tested across 13 diverse downstream datasets (such as PatchCamelyon, NCT-CRC, and SICAPv2) covering 8 different sub-pathologies.

Key Insights and Ideas

  • YouTube as an Expert Resource: While the general internet is noisy, educational videos from expert clinicians provide a high-density signal of "expert-narrated" data that is far more descriptive than standard medical reports.
  • LLMs as Medical Editors: The paper demonstrates that LLMs are highly effective at "denoising" specialized ASR output. By conditioning the LLM with medical ontologies, they achieved a sub-pathology classification accuracy of nearly 95%.
  • Beyond Single Labels: The results suggest that multiple descriptive sentences per image (averaging 1.74 per image in QUILT) foster a much more comprehensive understanding of tissue characteristics than a single class label.
Figure 2: QUILT examples. Input is the corrected ASR caption for the representative image. Output are the medical and ROI extracted text(s) paired with the image (see Section 3.1). In histopathology, understanding tissue characteristics often involves views from varying magnification levels. Thus, in QUILT we estimate an image’s magnification (indicated by the relative size of the microscope icon).

Key Results

The authors fine-tuned a CLIP model on Quilt-1M to create QuiltNet.

  • Zero-Shot Superiority: QuiltNet outperformed state-of-the-art models like PLIP and BiomedCLIP on 10 out of 12 zero-shot classification tasks. On the LC25000 (Colon) dataset, QuiltNet achieved 93.28% accuracy, compared to PLIP's 77.79%.
  • Linear Probing: Even when the model was allowed to "see" a small amount of labeled data (1% or 10% shots), QuiltNet consistently maintained a lead, often surpassing fully supervised models with only a fraction of the labels.
  • Cross-Modal Retrieval: In tasks like finding the right image for a given medical description, QuiltNet showed significant gains, particularly in the R@50 and R@200 metrics, proving it has learned a robust joint embedding space for pathology.

Limitations and Future Work

  • ASR Fragility: Despite LLM correction, some ASR errors are unrectifiable if the original audio was too poor or the term was entirely missing from the ontology.
  • Demographic Bias: The dataset relies on English-language videos, which the authors acknowledge may introduce a "Western institution" bias in the types of pathologies and narrations represented.
  • Classifier Errors: The automated histopathology image classifier had a ~5% false-positive rate, meaning some non-pathology images might still exist in the 1M pairs.
  • Future Directions: The authors suggest exploring other self-supervised objectives beyond contrastive learning to further leverage the temporal nature of video data.

Summary updated

Abstract

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering $1,087$ hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of $802, 144$ image and text pairs. QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around $200$K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with $1$M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across $13$ diverse patch-level datasets of $8$ different sub-pathologies and cross-modal retrieval tasks.
BibTeX
@inproceedings{ikezogwo2023quilt,
  author = {Wisdom Oluchi Ikezogwo and Mehmet Saygin Seyfioglu and Fatemeh Ghezloo and Dylan Stefan Chan Geva and Fatwir Sheikh Mohammed and Pavan Kumar Anand and Ranjay Krishna and Linda Shapiro},
  title = {{Quilt-1M: One Million Image-Text Pairs for Histopathology}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=ku2j7GP0Ql},
}
Figure