Biomedical article retrieval using multimodal features and image annotations in region-based CBIR

Daekeun You, Sameer K. Antani, Dina Demner-Fushman, Md. Mahmudur Rahman, Venu Govindaraju, George R. ThomaDRR 2010 · DOI

TL;DR

This paper introduces a novel method for improving biomedical article retrieval by automatically annotating images with semantic information extracted from text. By identifying and localizing image pointers, the system can associate specific image regions with biomedical concepts, leading to more precise content-based image retrieval (CBIR) and better overall document retrieval.

What Problem Does This Paper Address?

Clinicians and researchers rely heavily on biomedical literature for decision support and staying updated. While text-based search engines like MEDLINE are powerful, they often fall short when crucial information is conveyed visually within images. Biomedical articles frequently use images to illustrate complex concepts or highlight specific cases, but these images are often difficult to retrieve effectively. Traditional CBIR systems primarily rely on low-level visual features (color, texture), leading to a "semantic gap" where high-level medical concepts are not well represented. This paper aims to bridge this gap by developing a system that can automatically annotate images with relevant biomedical concepts, thereby enhancing the retrieval of both images and the articles they belong to.

Methods

The core of this work lies in a multimodal approach that combines image analysis and natural language processing. The key steps include:

  • Pointer Recognition: A novel Markov Random Field (MRF)-based method is proposed for localizing and recognizing image annotations, specifically pointers (like arrows and arrowheads), which authors use to highlight regions of interest (ROIs). This method models pointer boundaries as a sequence of line segments and uses contextual dependencies between these segments. A Hidden Markov Model (HMM) is then employed to classify these recognized pointers into categories like straight arrows, curved arrows, and arrowheads.
  • Image Feature Extraction: Standard image features (MPEG7, LIRE, Gabor filters, dominant colors, wavelet transforms) are extracted from both the whole image and the identified ROIs.
  • Text Analysis and Concept Mapping: Textual information from article titles, abstracts, figure captions, and mentions is analyzed. Biomedical concepts are identified using the Unified Medical Language System (UMLS) Metathesaurus.
  • Multimodal Retrieval: The system integrates text-based retrieval with region-based CBIR. Text queries are used to retrieve relevant articles, and then the identified ROIs within these articles' images are used to re-rank the results. If an image query is provided, its features are directly compared with indexed image features.

Datasets

  • Training and Testing Data: The authors utilized images from the ImageCLEFmed dataset, specifically cropping pointer images for training and testing the pointer recognition algorithm.
  • Evaluation Datasets: For the retrieval experiments, two topics from ImageCLEFmed were used: Topic 11 (abdominal CT images showing liver blood vessels) and Topic 21 (photographs of tumors).

Key Insights and Ideas

  • Bridging the Semantic Gap: The primary insight is that by automatically annotating image regions with semantic concepts derived from text, the "semantic gap" in CBIR can be significantly reduced, leading to more meaningful retrieval.
  • Pointers as Semantic Anchors: Image pointers are identified as crucial visual cues that authors use to direct attention to important image content. Recognizing these pointers allows for precise localization of ROIs.
  • MRF-HMM for Robust Pointer Recognition: The combination of MRF for detailed boundary labeling and HMM for classification proves effective in handling the variability and complexity of pointer shapes in biomedical images.
  • Multimodal Synergy: The paper demonstrates that combining text-based and image-based retrieval, especially with semantically annotated image regions, yields better results than either approach alone.

Key Results

  • Pointer Recognition: The MRF-HMM method achieved an overall accuracy of 82.0% in correctly labeling and classifying pointer boundaries from test images.
  • Biomedical Image Retrieval:
    • For Topic 21 (tumors), the proposed retrieval approach showed improvement, with six new relevant images ranked within the top-20 after re-ranking, compared to only two in the text-based retrieval.
  • For Topic 11 (liver blood vessels), the mean average precision (MAP) score was lower with the proposed approach. This was attributed to issues like images lacking pointers, pointer recognition failures, and inaccurate ROI extraction.

Limitations and Future Work

  • Limitations:
    • The pointer recognition algorithm still faces challenges with certain types of pointers and can be affected by errors in pre-processing (e.g., broken boundaries).
  • Inaccurate ROI extraction was identified as a significant factor limiting retrieval performance in some cases.
  • The system's effectiveness can be reduced in images that do not contain pointers.
  • Future Work:
    • Improving ROI extraction techniques is a key area for future development.
  • Further enhancing the MRF labeling algorithm and training the recognizer with a wider variety of unknown pointer parts.
  • Integrating text analysis results more deeply with pointer recognition and ROI extraction.
  • Automating processes currently done manually, such as noise ROI removal.
  • Conducting more extensive tests on mixed collections and diverse query topics.

Summary updated

BibTeX
@inproceedings{you2010biomedical,
  author = {Daekeun You and Sameer K. Antani and Dina Demner-Fushman and Md. Mahmudur Rahman and Venu Govindaraju and George R. Thoma},
  title = {{Biomedical article retrieval using multimodal features and image annotations in region-based CBIR}},
  booktitle = {DRR},
  year = {2010},
  doi = {10.1117/12.838973},
}
Figure