Visual Question Generation from Radiology Images

Mourad Sarrouti, Asma Ben Abacha, Dina Demner-FushmanACL 2020

TL;DR

This paper introduces VQGR, the first system to automatically generate natural language questions from radiology images. By employing a variational autoencoder (VAE) architecture and novel data augmentation techniques, VQGR addresses the critical challenge of limited labeled data in medical visual question answering (VQA), paving the way for more efficient VQA dataset creation and improved medical education tools.

What Problem Does This Paper Address?

The field of Visual Question Generation (VQG) aims to automatically create questions based on image content. While VQG has seen progress in general domains, its application to medical imaging, specifically radiology, has been largely unexplored. This is primarily due to the significant challenge of acquiring large, labeled datasets for medical VQA, which are essential for training supervised learning models. The authors highlight that existing medical VQA datasets are too small to effectively train VQG systems. This lack of data hinders the development of tools that could benefit patients by helping them understand medical images, support medical education, and aid in clinical decision-making.

Methods

The proposed VQGR system utilizes a Variational Autoencoder (VAE) architecture. The VAE consists of two main components:

  • Encoder: A Convolutional Neural Network (CNN) processes the radiology image to extract a feature map, which is then encoded into a latent representation space. This latent space captures the essential information from the image.
  • Decoder: A Long Short-Term Memory (LSTM) network takes a sample from the latent space and generates a natural language question word by word, starting with a "start" symbol and ending with an "end" symbol.

To overcome the data scarcity issue, VQGR employs two key data augmentation strategies:

  • Question Augmentation: Existing questions from the VQA-RAD dataset are expanded by replacing nouns and verbs with contextually similar words. These similar words are identified using pre-trained Wiki-PubMed-PMC embeddings based on cosine similarity. The authors found that replacing words with their top 20 most similar counterparts yielded the best results.
  • Image Augmentation: Standard image augmentation techniques such as flipping, rotation, shifting, and blurring are applied to the training images.

The overall VQGR model is trained by minimizing a combined loss function that includes:

  • Reconstruction Loss: Measures how well the VAE can reconstruct the input image from its latent representation.
  • Kullback-Leibler (KL) Divergence: Acts as a regularizer, ensuring the latent space distribution is close to a prior distribution (assumed to be a multivariate Gaussian).
  • Question Generation Loss: A Cross-Entropy loss that measures the difference between the generated question and the ground truth question.

Datasets

The study uses the VQA-RAD dataset for both training and evaluation. This dataset comprises 315 radiology images and 3,515 corresponding questions. The authors focus on questions related to "Modality," "Abnormality," "Organ," and "Plane." The VQA-RAD dataset is noted as being small, which is the primary motivation for the data augmentation techniques employed.

Key Insights and Ideas

  • Medical VQG is a Novel Frontier: The paper establishes that generating questions from medical images is an under-explored area, highlighting a significant research gap.
  • Data Augmentation is Crucial: The success of VQGR is heavily attributed to the data augmentation techniques, which effectively address the data scarcity problem in medical VQG. This suggests that data augmentation can be a powerful tool for developing models in data-limited medical AI tasks.
  • VAE for Image-to-Text Generation: The use of VAEs, specifically combining CNNs for image encoding and LSTMs for text decoding, proves effective for generating coherent and relevant questions from visual input.
  • Beyond Trivial Questions: The paper emphasizes the goal of generating "non-trivial" questions that require actual understanding of image content, moving beyond generic questions like "what is the abnormality in this image?".

Key Results

  • Automatic Evaluation: VQGR significantly outperforms a baseline system (trained on the original VQA-RAD dataset without augmentation) across multiple language modeling metrics, including BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, ROUGE-L, and CIDEr. For instance, VQGR achieved a CIDEr score of 31.1 compared to the baseline's 21.1.
  • Human Evaluation: Manual evaluations by two experts rated VQGR questions on relevancy, grammaticality, and fluency. VQGR achieved high scores (e.g., 93.3 for grammaticality), demonstrating its ability to generate human-like questions. The authors note that the baseline system consistently generated the same trivial question, making its human evaluation results less informative.
  • Inter-rater Reliability: The inter-rater reliability scores, while not exceptionally high (e.g., 0.42 for relevancy), are considered satisfactory, indicating reasonable agreement among human evaluators.

Limitations and Future Work

  • Dataset Size: Despite data augmentation, the VQA-RAD dataset is still relatively small, which could limit the model's ability to generalize to a wider range of medical images and question types.
  • Trivial Questions from Baseline: The baseline system's inability to generate diverse questions is a limitation of the comparison, though it highlights VQGR's advantage.
  • Future Work: The authors plan to investigate the use of generated questions to advance Visual Question Answering (VQA) in the medical domain. This suggests a future direction of leveraging VQGR to create better VQA datasets or improve VQA model training.

Summary updated

BibTeX
@inproceedings{sarrouti2020visual,
  author = {Mourad Sarrouti and Asma Ben Abacha and Dina Demner-Fushman},
  title = {{Visual Question Generation from Radiology Images}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure