Extraction Meets Abstraction: Ideal Answer Generation for Biomedical Questions

Yutong Li, Nicholas Gekakis, Qiuze Wu, Boyue Li, Khyathi Chandu, Eric NybergEMNLP 2018

TL;DR

This paper tackles the challenge of generating human-readable answers to biomedical questions by moving beyond simple extractive summarization. They introduce novel sentence ordering and fusion techniques, combined with a systematic configuration exploration framework, to create more coherent and abstractive answers, significantly improving human readability while maintaining competitive ROUGE scores.

What Problem Does This Paper Address?

Biomedical research is exploding, leading to a massive volume of publications. Researchers need efficient ways to find answers to specific questions within this deluge of information. While existing Biomedical Question Answering (BQA) systems can extract relevant sentences, they often produce answers that are incoherent, grammatically awkward, and difficult for humans to read. This is because simply stitching together the most relevant sentences from different documents can lead to abrupt topic shifts and redundant phrasing. The paper aims to bridge this gap by developing BQA systems that generate answers that are not only accurate but also human-readable and abstractive, mimicking how a human expert would synthesize information.

Methods

The core of this work lies in enhancing a baseline extractive BQA system. The authors introduce three main improvements:

  • Sentence Ordering: To address the issue of answers jumping between topics, they explore three sentence ordering algorithms:
  • Similarity Ordering: Groups similar sentences together.
  • Majority Ordering: Groups sentences from the same source document and maintains their original order.
  • Block Ordering: A hybrid approach that groups sentences by document and then orders these blocks based on length and similarity to the preceding sentence. This proved to be the most effective.
  • Sentence Fusion: To combat redundancy and improve abstraction, they employ a sentence fusion technique. This involves building a directed word graph from candidate sentences, identifying shared words and phrases, and then finding K-shortest paths to create new, more concise sentences. This process also includes removing transition words and filtering for readability.
  • Configuration Space Exploration (CSE): To systematically find the best combination of modules and hyperparameters, they utilize the BOOM framework. This allows them to explore thousands of configurations, optimizing for performance metrics like ROUGE.

Datasets

The paper uses data from the BioASQ challenge, specifically the sixth edition of Phase B. They evaluated their system on test batches from the BioASQ 4th edition. No new benchmarks were introduced; instead, they focused on improving performance on an existing competition dataset.

Key Insights and Ideas

  • Human Readability is Key: While ROUGE scores are important for automatic evaluation, they don't fully capture the quality of an answer for human consumption. The paper highlights that extractive systems, despite good ROUGE scores, often fail on human readability.
  • Block Ordering Outperforms: The Block Ordering algorithm, which combines document-based grouping with length and similarity-based ordering, significantly improved coherence compared to purely similarity-based or majority-based ordering.
  • Fusion for Abstraction: Sentence fusion is crucial for moving beyond simple extraction. It allows for the removal of redundant information and the creation of more synthesized, abstractive answers, even if it slightly impacts ROUGE scores.
  • Systematic Exploration is Powerful: The BOOM framework for Configuration Space Exploration is vital for optimizing complex systems with many modules and hyperparameters. It allows for a thorough search for optimal configurations that might be missed by manual tuning.

Key Results

  • The Block Ordering algorithm achieved a "reasonable" rating for 75% of answers in a manual evaluation, outperforming Similarity Ordering (55%) and Majority Ordering (71%).
  • The baseline system achieved a ROUGE-2 score of 0.6948.
  • After incorporating sentence fusion and ordering, the system achieved a ROUGE-2 score of 0.6257 and ROUGE-SU4 of 0.6214 on test batch 4 of BioASQ 4b. While the ROUGE score dropped slightly with fusion, the human readability was significantly improved.
  • Manual examination of fused sentences showed that 47% improved readability by reducing redundancy, though about 26% were found to be incoherent.

Limitations and Future Work

  • Fusion Quality Variability: The authors acknowledge that the quality of fused sentences is not always stable, with a significant portion (around 26%) being incoherent.
  • Opening Sentence Identification: The Block Ordering algorithm sometimes struggled to identify high-quality opening sentences, potentially due to relying on block length as a heuristic for topic relevance.
  • Manual Evaluation Cost: Performing manual evaluations is time-consuming, which is a challenge for tuning complex systems.
  • Future Work: The authors suggest exploring human evaluation metrics for sentence ordering, investigating further abstraction techniques beyond simple fusion, and developing methods to automatically predict reasonable sentence orderings. They also plan to refine the system architecture by moving sentence fusion before MMR to handle fused sentences more effectively.

Summary updated

BibTeX
@inproceedings{li2018extraction,
  author = {Yutong Li and Nicholas Gekakis and Qiuze Wu and Boyue Li and Khyathi Chandu and Eric Nyberg},
  title = {{Extraction Meets Abstraction: Ideal Answer Generation for Biomedical Questions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure