Similar image search for histopathology: SMILY

Narayan Hegde, Jason D. Hipp, Yun Liu, Michael Emmert-Buck, Emily Reif, Daniel Smilkov, Michael Terry, Carrie J. Cai, Mahul B. Amin, Craig H. Mermel, Philip Q. Nelson, Lily H. Peng, Gregory S. Corrado, Martin C. Stumpe

TL;DR

This paper introduces SMILY, a deep learning-based reverse image search tool for histopathology images. SMILY enables efficient searching of large digital pathology archives by finding visually similar image patches, significantly outperforming traditional methods and random search in retrieving images with similar histologic features, organ sites, and cancer grades.

What Problem Does This Paper Address?

Histopathology images are massive, often exceeding 100,000 pixels in each dimension. While these images are increasingly digitized and stored in large datasets, finding specific regions of interest (e.g., a particular cellular structure or lesion) within them is a laborious manual process. Existing metadata, like diagnoses or clinical notes, often applies to the entire image or set of images, lacking the granular detail needed for precise localization. This makes it challenging for pathologists, researchers, and trainees to efficiently search and analyze these vast image collections for diagnosis, research, or education.

Methods

The core of SMILY is a deep learning-based approach using a convolutional neural network (CNN) architecture called a "deep ranking network." This network is trained to generate numerical "embeddings" for image patches, effectively creating a compressed representation of their visual content. The key innovation here is that the network was not trained on histopathology images specifically, but rather on a large dataset of "natural images" (dogs, cats, objects, etc.). This pre-training allows the network to learn general visual similarity, which is then applied to histopathology.

When a user selects a query image patch, SMILY computes its embedding and then compares it to the embeddings of all patches in its database using L2 distance. To handle the orientation-independence of histopathology images, SMILY generates multiple orientations (rotations and mirrors) for each patch, creating eight embeddings per patch. For efficient searching across massive datasets, they employ k-d trees and parallelize computations across multiple machines.

Datasets

The primary dataset used for building the SMILY database and for evaluation was The Cancer Genome Atlas (TCGA). This publicly available resource contains a large collection of histopathology slides. For training the embedding network itself, a dataset of approximately 500,000,000 "natural images" from 18,000 distinct classes was used.

Key Insights and Ideas

  • General-purpose pre-training is effective: A significant insight is that a deep learning model pre-trained on diverse "natural" images can effectively learn visual similarity for histopathology without requiring specific histopathology training data. This bypasses the costly and time-consuming process of annotating large histopathology datasets.
  • "Similarity" is multifaceted: The paper highlights that "similarity" in histopathology can mean many things (e.g., same stain, same histologic feature, same organ). SMILY aims to capture a broad sense of similarity, returning results that match on multiple axes, which is more useful than narrowly defined similarity.
  • Efficiency for large-scale data: The technical challenges of handling terabytes of histopathology data are addressed through efficient embedding generation, storage, and optimized search algorithms (k-d trees, parallelization), enabling sub-second query times.

Key Results

SMILY demonstrated strong performance in retrieving similar histopathology images:

  • Histologic Feature Match: In prostate specimens, SMILY achieved a 63.9% top-5 score for retrieving images of the same histologic feature, significantly outperforming SIFT (45.2%) and random search (28.3%).
  • Prostate Cancer Gleason Grade: SMILY was more accurate than SIFT baselines in retrieving images with the correct Gleason patterns (76.0% vs. 65.2%).
  • Pathologist Evaluation: In blinded prospective studies, pathologists rated SMILY search results significantly higher than random search for histologic feature match (62.1% vs. 26.8%), organ site match (68.3% of results), and overall match quality for prostate cancer patches (61.0% vs. 30.0%).
  • Speed: Queries on a large TCGA dataset (10^9 patches) took a median of 1.3 seconds using optimized methods.

Limitations and Future Work

  • General-purpose vs. Application-specific: While SMILY is a powerful general-purpose tool, the authors acknowledge that application-specific tools might achieve higher accuracy for very niche tasks (e.g., precise mitotic counting).
  • Defining "Similarity": The paper discusses the ambiguity of "similarity" and suggests that future work could focus on enabling users to refine search results along specific axes of similarity.
  • Expanding Similarity Axes: SMILY was evaluated on histologic features, organ site, and prostate cancer grade. Further validation is needed for other potential axes of similarity.
  • Dataset Size and Diversity: While TCGA is large, increasing the number of slides could further improve the capture of diverse tissue processing conditions and resulting image appearances.
  • Refinement of Search Results: Future work could explore interactive refinement of search results to allow users to specify desired similarity criteria.

Summary updated

Abstract

The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Although these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. As pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep-learning-based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY’s ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist’s arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.
BibTeX
@inproceedings{hegde2019similar,
  author = {Narayan Hegde and Jason D. Hipp and Yun Liu and Michael Emmert-Buck and Emily Reif and Daniel Smilkov and Michael Terry and Carrie J. Cai and Mahul B. Amin and Craig H. Mermel and Philip Q. Nelson and Lily H. Peng and Gregory S. Corrado and Martin C. Stumpe},
  title = {{Similar image search for histopathology: SMILY}},
  booktitle = {Npj Digit. Medicine},
  year = {2019},
  url = {https://openreview.net/forum?id=3i9sjQvWQW1},
}
Figure