Bimodal Network Architectures for Automatic Generation of Image Annotation from Text

Mehdi Moradi, Ali Madani, Yaniv Gur, Yufan Guo, Tanveer F. Syeda-Mahmood

Abstract

Medical image analysis practitioners have embraced big data methodologies. This has created a need for large annotated datasets. The source of big data is typically large image collections and clinical reports recorded for these images. In many cases, however, building algorithms aimed at segmentation and detection of disease requires a training dataset with markings of the areas of interest on the image that match with the described anomalies. This process of annotation is expensive and needs the involvement of clinicians. In this work we propose two separate deep neural network architectures for automatic marking of a region of interest (ROI) on the image best representing a finding location, given a textual report or a set of keywords. One architecture consists of LSTM and CNN components and is trained end to end with images, matching text, and markings of ROIs for those images. The output layer estimates the coordinates of the vertices of a polygonal region. The second architecture uses a network pre-trained on a large dataset of the same image types for learning feature representations of the findings of interest. We show that for a variety of findings from chest X-ray images, both proposed architectures learn to estimate the ROI, as validated by clinical annotations. There is a clear advantage obtained from the architecture with pre-trained imaging network. The centroids of the ROIs marked by this network were on average at a distance equivalent to 5.1% of the image width from the centroids of the ground truth ROIs.

Introduction

The rapid advancement of medical AI is often hampered by the "annotation gap." While massive repositories of clinical images and corresponding text reports exist, training models for disease detection or segmentation requires precise ground-truth annotations—specifically, Regions of Interest (ROIs) that pinpoint abnormalities. Manually generating these annotations is a resource-intensive process requiring expert radiologists. This research explores whether deep learning can bridge this gap by automatically mapping textual descriptions in medical reports to specific pixel coordinates in chest X-rays.

Proposed Architectures

The researchers developed and compared two bimodal architectures designed to treat ROI generation as a coordinate regression problem, predicting the vertices of a four-sided polygon.

  1. Architecture 1 (End-to-End): This model is trained from scratch. It utilizes a standard Convolutional Neural Network (CNN) for image processing and a Long Short-Term Memory (LSTM) network to encode medical keywords (MeSH terms). The resulting bimodal features are concatenated and passed through a fully connected network to predict ROI coordinates.
  2. Architecture 2 (Pre-trained/Transfer Learning): This approach leverages transfer learning by using a DenseNet121 pre-trained on the NIH Chest-Ray14 dataset (112,000 images) as a feature generator. For text, it employs a Doc2Vec model trained on thousands of reports. These high-level features are then mapped to the ROI coordinates.
Architecture 2: A transfer learning approach using a pre-trained DenseNet121 for image features and Doc2Vec for text features.

Experimental Results

The models were evaluated using a subset of the Indiana University Hospital dataset, with 494 images manually annotated by a radiologist to serve as the gold standard. Architecture 2 significantly outperformed the end-to-end approach, achieving a centroid distance error of only 5.1% of the image width, compared to 7.2% for Architecture 1. This highlights the effectiveness of using large-scale, weakly labeled data to learn general anatomical features before fine-tuning on specific annotation tasks.

Qualitative results comparing ground truth (red) and predicted (blue) regions of interest across various chest X-ray findings.

Qualitatively, the models showed high proficiency in identifying findings like cardiomegaly (enlarged heart), consistently mapping the ROI to the heart silhouette. Furthermore, the inclusion of "normal" scans in the training set allowed the models to learn when not to generate a marking, returning [0,0] coordinates when the text indicated no findings. The researchers found that the text acts as a vital constraint; without it, error rates tripled, as the text provides the necessary context to prioritize specific visual features.

Conclusion

This work demonstrates that bimodal networks can effectively automate the generation of image annotations from clinical reports. Architecture 2 achieved an overall Dice coefficient of 61%, which is impressive given the approximate nature of human-drawn bounding boxes. Future work will focus on expanding the model to handle irregular shapes beyond quadrilaterals and optimizing for reports containing multiple distinct findings described in a single report.

Summary updated

Abstract

Medical image analysis practitioners have embraced big data methodologies. This has created a need for large annotated datasets. The source of big data is typically large image collections and clinical reports recorded for these images. In many cases, however, building algorithms aimed at segmentation and detection of disease requires a training dataset with markings of the areas of interest on the image that match with the described anomalies. This process of annotation is expensive and needs the involvement of clinicians. In this work we propose two separate deep neural network architectures for automatic marking of a region of interest (ROI) on the image best representing a finding location, given a textual report or a set of keywords. One architecture consists of LSTM and CNN components and is trained end to end with images, matching text, and markings of ROIs for those images. The output layer estimates the coordinates of the vertices of a polygonal region. The second architecture uses a network pre-trained on a large dataset of the same image types for learning feature representations of the findings of interest. We show that for a variety of findings from chest X-ray images, both proposed architectures learn to estimate the ROI, as validated by clinical annotations. There is a clear advantage obtained from the architecture with pre-trained imaging network. The centroids of the ROIs marked by this network were on average at a distance equivalent to 5.1% of the image width from the centroids of the ground truth ROIs.
BibTeX
@inproceedings{moradi2018bimodal,
  author = {Mehdi Moradi and Ali Madani and Yaniv Gur and Yufan Guo and Tanveer F. Syeda-Mahmood},
  title = {{Bimodal Network Architectures for Automatic Generation of Image Annotation from Text}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  doi = {10.1007/978-3-030-00928-1\_51},
}
Figure