Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation

He Zhu, Ren Togo, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel medical visual question generation (VQG) model that leverages a multi-latent space approach to create diverse and informative questions from medical images. By employing a new diversity learning strategy and a custom "Similarity" metric, the model aims to improve automated clinical diagnosis by generating a richer set of questions than previous methods.

What Problem Does This Paper Address?

The paper tackles a critical challenge in healthcare: the uneven distribution of clinical resources and the heavy reliance on human physicians for diagnosis. While automated clinical diagnosis systems, particularly those involving question-answering (QA) and visual question generation (VQG), show promise, current VQG models often fail to generate a sufficiently diverse set of questions from a single medical image. This lack of diversity limits the system's ability to capture a patient's condition from multiple perspectives, hindering accurate diagnosis. Existing evaluation metrics also struggle to adequately assess the diversity and semantic richness of generated questions.

Methods

The core of the proposed method is a diversity learning-based VQG model utilizing a multi-latent space. This approach builds upon the encoder-decoder architecture common in VQG.

  • Image-Category Feature Processor: This component integrates image features with category information (e.g., "Abnormality," "Organ") to allow for controllable question generation.
  • Multi-Latent Space Processor: Instead of a single latent space, the model employs multiple latent spaces (K). Each space is designed to capture different aspects of the input image and category information. This is achieved by learning the means and standard deviations of multivariate Gaussian distributions for each latent space.
  • Diversity Learning: To ensure these latent spaces learn distinct information, the model minimizes a Maximal Information Coefficient (MIC) loss. This loss function encourages dissimilarity between the latent spaces, thereby promoting diverse information capture.
  • Question Set Generation: A Long Short-Term Memory (LSTM) decoder is used to generate question sets from the learned latent spaces. Crucially, the model generates entire question sets simultaneously, rather than one question at a time, to avoid repetition and further enhance diversity.
  • New Evaluation Metric: Similarity: To address the shortcomings of traditional metrics like BLEU and METEOR, which can be misled by word overlap rather than semantic meaning, the authors introduce a "Similarity" metric. This metric uses the cosine similarity of sentence embeddings (from Google's universal sentence-encoder) to quantify the semantic relatedness between generated questions and ground truth questions. Lower Similarity scores indicate greater diversity and informativeness.

Datasets

The experiments were conducted using two medical imaging datasets:

  • VQA-RAD: A manually constructed medical VQA dataset containing radiology images with clinician-created questions and answers.
  • Slake: A dataset that includes healthy and unhealthy cases from various sources, with expert physicians labeling organs and diseases.

The datasets were preprocessed to select six categories of QA pairs: "Abnormality," "Modality," "Organ," "Plane," "Position," and "Others."

Key Insights and Ideas

  • Multi-Latent Space for Diversity: The core idea that different latent spaces can learn distinct, complementary information from the same input is powerful. This allows for a more comprehensive understanding of the image and, consequently, the generation of more varied questions.
  • MIC Loss for Explicit Diversity: Explicitly training for dissimilarity between latent spaces using MIC loss is a novel way to enforce diversity, moving beyond implicit diversity achieved through sampling.
  • Simultaneous Question Set Generation: Generating entire sets of questions at once, rather than iteratively, is a practical approach to reduce redundancy and increase the overall diversity of the output.
  • "Similarity" Metric for Semantic Evaluation: The introduction of a new metric that focuses on semantic similarity (or lack thereof) is a significant contribution, as it directly addresses the limitations of existing metrics in evaluating VQG diversity.

Key Results

  • Improved Diversity: The proposed multi-latent space model, particularly with 3 latent spaces, significantly outperformed previous state-of-the-art models in terms of diversity, as measured by the new "Similarity" metric. The average Similarity score was notably lower for the proposed model compared to others.
  • Quantitative Performance: While the primary focus is diversity, the model also showed competitive performance on traditional language modeling metrics like BLEU, METEOR, ROUGE-L, and CIDEr. For instance, "Our model-3" achieved the highest scores in several of these metrics.
  • Optimal Number of Latent Spaces: Experiments with 2, 3, and 4 latent spaces indicated that there's an optimal number of latent spaces for a given dataset and model complexity. For the tested datasets, 3 latent spaces yielded the best results, suggesting that too many spaces can lead to diminishing returns or difficulty in controlling diversity.
  • Ablation Study: An ablation study confirmed the effectiveness of the newly proposed MIC loss, showing that its inclusion significantly improved the model's accuracy.

Limitations and Future Work

  • Limitations:
    • Accuracy: The authors acknowledge that the model's accuracy needs to be higher, especially compared to models trained on real-scene datasets. This is attributed to the specialized vocabulary and smaller size of medical datasets, making image understanding more challenging.
  • Duplicate Questions: Despite efforts, some duplicate questions are still generated.
  • Category Control: Controlling question categories is more difficult in the medical domain due to overlapping and ambiguous classifications.
  • Optimal Latent Space Number: The optimal number of latent spaces is dataset-dependent and requires tuning.
  • Future Work:
    • Expand Dataset and Improve Accuracy: Continue to increase dataset size and explore techniques for higher accuracy.
  • Leverage Large-Scale Pre-trained Models: Fine-tune transformer-based models (like BERT and large image models) for improved performance.
  • Incorporate External Knowledge: Integrate external knowledge sources to enhance question generation and personalization.
  • Enhance Visual Reasoning: Improve the model's ability to understand images by incorporating semantic features.
  • Deploy on Robots: Explore deploying the interactive QA system on robots for more flexible decision-making in complex environments.

Summary updated

Abstract

Auxiliary clinical diagnosis has been researched to solve unevenly and insufficiently distributed clinical resources. However, auxiliary diagnosis is still dominated by human physicians, and how to make intelligent systems more involved in the diagnosis process is gradually becoming a concern. An interactive automated clinical diagnosis with a question-answering system and a question generation system can capture a patient’s conditions from multiple perspectives with less physician involvement by asking different questions to drive and guide the diagnosis. This clinical diagnosis process requires diverse information to evaluate a patient from different perspectives to obtain an accurate diagnosis. Recently proposed medical question generation systems have not considered diversity. Thus, we propose a diversity learning-based visual question generation model using a multi-latent space to generate informative question sets from medical images. The proposed method generates various questions by embedding visual and language information in different latent spaces, whose diversity is trained by our newly proposed loss. We have also added control over the categories of generated questions, making the generated questions directional. Furthermore, we use a new metric named similarity to accurately evaluate the proposed model’s performance. The experimental results on the Slake and VQA-RAD datasets demonstrate that the proposed method can generate questions with diverse information. Our model works with an answering model for interactive automated clinical diagnosis and generates datasets to replace the process of annotation that incurs huge labor costs.
BibTeX
@inproceedings{zhu2023diversity,
  author = {He Zhu and Ren Togo and Takahiro Ogawa and Miki Haseyama},
  title = {{Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=e-5QSoSQkPV},
}
Figure