Text- and Content-Based Medical Image Retrieval in the VISCERAL Retrieval Benchmark

Fan Zhang, Yang Song, Weidong Cai, Adrien Depeursinge, Henning Müller, Fan Zhang 0013, Yang Song 0001Cloud-Based Benchmarking of Medical Image Analysis 2017 · DOI · DBLP

Introduction

Efficiently retrieving relevant clinical cases from large-scale medical databases is a cornerstone of evidence-based medicine. Clinicians often seek "cases like this one" to aid in diagnosis and treatment planning. Traditionally, this has relied on text-based retrieval using manual annotations and radiology reports. However, manual labeling is labor-intensive and prone to inter-observer variability. Content-based image retrieval (CBIR) offers an automated alternative by extracting visual features directly from pixels. Yet, CBIR faces the "semantic gap"—the difficulty in mapping low-level visual patterns (e.g., texture) to high-level clinical concepts (e.g., a specific pathology). This paper evaluates multi-modal fusion strategies within the VISCERAL (Visual Concept Extraction and Retrieval in Radiology) benchmark to bridge this gap.

Methodology

The study explores five retrieval strategies, ranging from unimodal text or visual approaches to a combined multi-modal framework.

Text-Based Retrieval

Two text-based methods were implemented using anatomy and pathology terms mapped to the RadLex ontology. While TF-IDF provides a standard baseline, the researchers utilized Probabilistic Latent Semantic Analysis (pLSA) to discover hidden "topics" within the medical terms. This allows the system to identify similarities even when different terminology is used to describe the same underlying condition.

The pLSA process maps cases and terms to a latent topic space to capture semantic relationships.

Content-Based Image Retrieval (CBIR)

The visual retrieval pipeline utilizes a "Bag of Visual Words" (BoVW) approach. Local SIFT descriptors are extracted from 3D CT and MRI volumes and clustered into a visual dictionary. To improve performance, a Result Refinement step was introduced. This iterative ranking algorithm leverages the "neighborhood" of images; if multiple images similar to the query are also similar to each other, their relevance scores are boosted, filtering out visual outliers that lack clinical consistency.

The retrieval workflow integrates feature extraction, similarity computation, and result refinement.

Fusion Retrieval

The final strategy involves late fusion, where scores from the text-based and visual-based methods are normalized and combined using a weighted sum to produce the final ranked list.

Experimental Setup

The evaluation used the VISCERAL Retrieval dataset, comprising 2,311 3D volumes across CT, T1-weighted MRI, and T2-weighted MRI modalities. The benchmark included 10 clinical queries, each defined by a diagnosis, a case description, and a 3D bounding box for the Region of Interest (ROI).

Results and Discussion

Performance was assessed using Mean Average Precision (MAP) and Precision at 10 (P@10).

Method P@10 MAP bpref
TF-IDF (Text) 0.370 0.081 0.162
pLSA (Text) 0.410 0.094 0.183
BoVW (Visual) 0.250 0.078 0.190
Refinement (Visual) 0.330 0.083 0.188
Fusion (Combined) 0.420 0.110 0.207

The results indicate that pLSA significantly outperforms simple TF-IDF, highlighting the importance of latent semantic modeling. In the visual domain, the refinement step provided a notable boost to P@10. Ultimately, the multi-modal Fusion approach yielded the best overall performance (MAP 0.110), demonstrating that visual features and textual reports provide complementary information.

One observed limitation was the "whole-volume" approach for visual features. For queries involving small pathologies, global features were often dominated by surrounding healthy tissue, suggesting that future work should focus on localized feature extraction within 3D ROIs.

Conclusion

Combining textual semantics with refined visual features provides a robust framework for medical image retrieval. While the semantic gap remains a challenge, the use of latent topic modeling and neighborhood-based refinement significantly improves retrieval precision. Future work will explore early fusion techniques to more deeply integrate textual and visual data during the feature extraction phase.

Summary updated

BibTeX
@inproceedings{zhang2017text,
  author = {Fan Zhang and Yang Song and Weidong Cai and Adrien Depeursinge and Henning Müller and Fan Zhang and Yang Song},
  title = {{Text- and Content-Based Medical Image Retrieval in the VISCERAL Retrieval Benchmark}},
  booktitle = {Cloud-Based Benchmarking of Medical Image Analysis},
  year = {2017},
  doi = {10.1007/978-3-319-49644-3\_14},
}
Figure