Quilt-1M: One Million Image-Text Pairs for Histopathology

Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, Linda G. Shapiro

TL;DR

Researchers have created QUILT-1M, the largest-ever dataset of one million image-text pairs specifically for histopathology. This massive dataset, curated from YouTube videos and other sources, aims to bridge the gap in medical vision-language data and significantly advance AI models for analyzing microscopic tissue images.

What Problem Does This Paper Address?

The field of histopathology, crucial for diagnosing diseases by examining tissue samples under a microscope, has lagged behind other areas like natural image analysis in developing advanced AI models. A major bottleneck is the scarcity of large-scale, high-quality datasets that pair microscopic images with descriptive text. While general vision-language models have made strides, they often lack the specialized knowledge and nuanced descriptions needed for accurate histopathology analysis. Existing medical datasets are either too small, not openly available, or not specifically tailored for histopathology, hindering progress in areas like automated diagnosis and image-text retrieval.

Methods

The authors introduce QUILT, a large-scale vision-language dataset for histopathology, built from over 1,000 hours of educational YouTube videos. The curation process involved a sophisticated pipeline:

  • Video Sourcing: Identifying relevant histopathology videos using keywords and filtering for "narrative style" content where presenters explain images.
  • Image and Text Extraction: Extracting keyframes from videos and using Automatic Speech Recognition (ASR) to transcribe the audio.
  • Denoising and Refinement: Employing a multi-step process involving LLMs (like GPT-3.5), UMLS (Unified Medical Language System), and handcrafted algorithms to correct ASR errors, extract medically relevant text (MED), and identify specific regions of interest (ROI).
  • Alignment: Developing unique algorithms to align extracted image frames with their corresponding text descriptions, ensuring accurate pairings.
  • Dataset Expansion: Combining the curated QUILT dataset with other sources like PubMed articles, Twitter, and the LAION dataset to create QUILT-1M, a dataset of one million image-text pairs.

Datasets

  • QUILT: A newly curated dataset of 802,144 image-text pairs derived from 1,087 hours of YouTube histopathology videos.
  • QUILT-1M: The expanded dataset, comprising one million image-text pairs, created by merging QUILT with data from PubMed, Twitter, and LAION.
  • Downstream Evaluation Datasets: The authors evaluated their models on 13 diverse, publicly available histopathology datasets, including PatchCamelyon, NCT-CRC-HE-100K, SICAPv2, Databiox, BACH, Osteo, RenalCell, and SkinCancer, covering 8 different sub-pathologies.

Key Insights and Ideas

  • YouTube as an Untapped Resource: The paper highlights the immense potential of publicly available educational videos as a rich source for creating specialized medical datasets, overcoming the limitations of traditional data collection methods.
  • LLMs as Powerful Curation Tools: The successful use of LLMs for ASR error correction, text denoising, and sub-pathology classification demonstrates their significant role in automating and improving the quality of complex data curation pipelines.
  • The Importance of Dense, Interconnected Representations: The authors argue that single labels are insufficient for histopathology. Natural language descriptions provide a more comprehensive signal, linking diverse features of sub-patch structures, which is crucial for advanced AI models.
  • QUILT-1M's Scale is a Game-Changer: Creating the largest histopathology vision-language dataset to date is expected to unlock new possibilities for training more robust and accurate AI models in this domain.

Key Results

  • QUILTNET Performance: Fine-tuning a pre-trained CLIP model on QUILT-1M (named QUILTNET) significantly outperformed state-of-the-art models on various tasks.
  • Zero-Shot Classification: QUILTNET achieved superior performance across 12 zero-shot tasks, covering 8 different sub-pathologies, outperforming models like CLIP, PLIP, and BiomedCLIP.
  • Linear Probing: QUILTNET demonstrated strong few-shot and full-shot performance, even surpassing supervised models on some datasets with as little as 1% of the training data.
  • Cross-Modal Retrieval: The model showed competitive results in both text-to-image and image-to-text retrieval tasks on the QUILT-1M holdout set and the ARCH dataset.

Limitations and Future Work

  • Dataset Biases: The authors acknowledge that the dataset, derived from English-language YouTube videos, may exhibit biases, potentially performing better on data from Western demographics and underperforming on data from other cultural or linguistic groups.
  • ASR and LLM Errors: Despite extensive efforts, ASR can still misinterpret medical terms, and LLM corrections might occasionally lead to incorrect entries. The dataset is not entirely free of noise.
  • Limited Data for Training from Scratch: The paper suggests that while fine-tuning a pre-trained CLIP model was effective, training a CLIP model from scratch on QUILT-1M underperformed, indicating that even one million pairs might not be sufficient for robust scratch training.
  • Future Work: The authors suggest exploring other self-supervised objectives and acknowledge that future works may explore more advanced methods to address the limitations. They also plan to release the dataset and code to enable further research.

Summary updated

Abstract

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering $1,087$ hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of $802, 144$ image and text pairs. QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around $200$K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with $1$M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across $13$ diverse patch-level datasets of $8$ different sub-pathologies and cross-modal retrieval tasks.
BibTeX
@inproceedings{ikezogwo2023quilt,
  author = {Wisdom Oluchi Ikezogwo and Mehmet Saygin Seyfioglu and Fatemeh Ghezloo and Dylan Stefan Chan Geva and Fatwir Sheikh Mohammed and Pavan Kumar Anand and Ranjay Krishna and Linda G. Shapiro},
  title = {{Quilt-1M: One Million Image-Text Pairs for Histopathology}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=ku2j7GP0Ql},
}
Figure