Ontology-Based Retrieval & Neural Approaches for BioASQ Ideal Answer Generation

Ashwin Naresh Kumar, Harini Kesavamoorthy, Madhura Das, Pramati Kalwad, Khyathi Chandu, Teruko Mitamura, Eric NybergEMNLP 2018

TL;DR

This paper presents a novel system for generating concise, human-readable answers to biomedical questions by combining ontology-based information retrieval with neural learning-to-rank and abstractive summarization techniques. Their approach achieved the highest ROUGE score on the BioASQ 5b challenge, demonstrating a significant step forward in automated biomedical question answering.

What Problem Does This Paper Address?

Finding precise answers within the ever-growing volume of biomedical literature is a significant challenge for researchers. Traditional search engines often return too much information, making it difficult to pinpoint the exact knowledge needed. Biomedical Question Answering (BioQA) systems aim to automate this process by identifying relevant documents and extracting specific answers. The "Ideal Answer Generation" task, a part of the BioASQ challenge, focuses on creating non-redundant, human-readable summaries that directly address a given biomedical question. This paper tackles the challenge of improving the accuracy and readability of these generated answers, especially in the complex biomedical domain.

Methods

The authors propose a multi-stage pipeline for ideal answer generation:

  • Ontology-Based Information Retrieval: This module aims to enhance the retrieval of relevant information by leveraging biomedical ontologies. It extracts medical entities and their relationships from text, building a graphical representation. This structured knowledge is then used to query and retrieve relevant snippets, going beyond simple keyword matching.
  • Neural Learning-to-Rank (LETOR): After initial retrieval, a ranking module is employed to order the retrieved snippets by relevance to the question. They explored two LETOR approaches:
  • RankSVM: A pairwise learning-to-rank algorithm.
  • Neural Ranking Approach: A list-wise approach using a two-layer Multi-Layer Perceptron (MLP) inspired by ListNet. This model is trained to predict BM25 scores, aiming to estimate relevance rather than just relative ranking.
  • Feature Engineering: Extensive features were engineered for the LETOR models, including statistical (e.g., BM25 score, snippet length), semantic (e.g., word2vec embeddings), and syntactic features (e.g., medical entity counts).
  • Abstractive Summarization: The final stage uses a neural sequence-to-sequence model (Pointer-Generator Coverage network, PGC) to generate the ideal answer. They leveraged a model pretrained on general news summarization (CNN/Daily Mail) and then fine-tuned it on the BioASQ dataset to adapt it to the biomedical domain. This approach allows for generating novel sentences rather than just extracting existing ones, aiming for better coherence and readability.

Datasets

The primary dataset used for development and evaluation is from the BioASQ 5b training data. This dataset consists of 2,251 questions, each with associated relevant documents and snippets. The authors also mention using the CNN/Daily Mail dataset for pretraining their abstractive summarization model.

Key Insights and Ideas

  • Combining Extractive and Abstractive Techniques: The authors hypothesize that a hybrid approach, combining the strengths of both extractive (grammatically robust) and abstractive (coherent and readable) summarization, would yield better results than either method alone.
  • Leveraging Ontologies for Retrieval: Using biomedical ontologies to represent entities and relations significantly enhances the retrieval of relevant information, especially for complex biomedical queries.
  • Neural Ranking Outperforms Traditional Methods: The neural ranking approach, particularly when fine-tuned with domain-specific features, demonstrated superior performance compared to traditional methods like RankSVM and even BM25 alone.
  • Transfer Learning for Biomedical Summarization: Pretraining an abstractive summarization model on a large general domain dataset (CNN/Daily Mail) and then fine-tuning it on the biomedical BioASQ data proved effective, allowing the model to handle domain-specific vocabulary and generate relevant summaries.
  • Feature Importance: Statistical features, particularly BM25 and its logarithmic transformation, were found to be highly influential in the ranking process.

Key Results

The system achieved the highest ROUGE score of 0.659 on the BioASQ 5b batch 2 test. This score represents a significant improvement in the quality of generated ideal answers. The fine-tuned PGC model for summarization generated answers with higher ROUGE-2 and ROUGE-SU4 scores compared to the pretrained model. The neural ranking approach also outperformed RankSVM.

Limitations and Future Work

  • Limitations:
    • The ontology-based retrieval might return too many abstracts if a medical entity is mentioned frequently, requiring the ranking module to filter them.
  • For certain question types (e.g., summary type), there might not be a direct mapping between the ideal answer and the retrieved snippets, leading to incomplete answers.
  • Qualitative analysis revealed that while abstractive models avoid repetition, they might still define terms in a circular manner.
  • Future Work:
    • Explore other diversification algorithms for sentence selection beyond MMR.
  • Investigate more language model-based features for the LETOR pipeline, such as Pointwise Mutual Information.
  • Incorporate anaphora resolution to improve relation extraction.
  • Extend the ontology indexing and retrieval system to handle factoid and list-type questions.
  • Leverage entire documents, not just abstracts, for training end-to-end summarization networks.
  • Use question type as contextual information to improve answer precision.

Summary updated

BibTeX
@inproceedings{kumar2018ontology,
  author = {Ashwin Naresh Kumar and Harini Kesavamoorthy and Madhura Das and Pramati Kalwad and Khyathi Chandu and Teruko Mitamura and Eric Nyberg},
  title = {{Ontology-Based Retrieval \& Neural Approaches for BioASQ Ideal Answer Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure