Tackling Biomedical Text Summarization: OAQA at BioASQ 5B

Khyathi Raghavi Chandu, Aakanksha Naik, Aditya Chandrasekar, Zi Yang, Niloy Gupta, Eric NybergBioNLP 2017 · OpenReview

Abstract

Generating "ideal answers" to biomedical questions requires synthesizing concise, accurate summaries from vast amounts of technical literature. Researchers from Carnegie Mellon University developed an extractive summarization system for the BioASQ challenge that leverages a novel word-embedding-based similarity metric and "soft positional constraints." Their pipeline achieved top-tier performance by effectively bridging the semantic gap between queries and source documents.

Introduction

The exponential growth of the MEDLINE database, which now contains over 23 million references, presents a significant challenge for medical researchers. While traditional search engines can retrieve relevant documents, the task of Ideal Answer Generation—creating a concise, non-redundant, and readable summary that directly answers a specific query—remains difficult. This paper describes the OAQA system's approach to Phase B of the BioASQ challenge, focusing on the semantic understanding required to link questions with answer sentences across multiple documents.

Methodology

The OAQA system employs a three-stage pipeline: Relevance Ranking, Sentence Selection, and Sentence Tiling.

The OAQA system architecture for biomedical question answering and summarization.

Key Components:

  1. Granularity and Filtering: The team found that using pre-filtered "snippets" provided by the challenge yielded better results than extracting sentences from full PubMed abstracts, as snippets are already high-relevance candidates.
  2. w2v tf-idf Similarity: To move beyond exact keyword matching, the authors integrated Word2Vec embeddings (pre-trained on PubMed and Wikipedia) into a tf-idf framework. This allows the system to recognize semantic relationships between different terms, such as a drug and its therapeutic class.
  3. Soft Positional Constraints: Recognizing that the order of snippets often reflects their importance, a weighting parameter ($\alpha$) was introduced to boost the scores of sentences appearing earlier in the source list.
  4. Maximal Marginal Relevance (MMR): MMR was utilized to balance query relevance with information novelty, effectively reducing redundancy in the final summary.
  5. Sentence Compression: An LSTM-based neural network was tested to remove redundant introductory phrases (e.g., "We report that..."), improving readability.

Results and Evaluation

The system was evaluated on the BioASQ 4 and 5B datasets. The combination of MMR, soft positional constraints, and w2v tf-idf similarity proved to be the most effective configuration.

Performance comparison of different experimental configurations using ROUGE metrics.

As shown in the results table, the "Soft positional constraint + w2v tf-idf similarity" configuration (Experiment 17) achieved the highest scores, with a ROUGE-2 of 0.6534 and a ROUGE-SU4 of 0.6536. This significantly outperformed the "first snippet" baseline (Experiment 12), which scored only 0.3363 in ROUGE-2.

Discussion

A key insight from the study was the value of the "position signal" in the source snippets. Simply favoring earlier content provided a measurable performance boost. Conversely, while medical ontologies like UMLS and SNOMEDCT are valuable, "unbounded" expansion often introduced noise that hindered performance. This suggests that controlled semantic similarity via embeddings is more robust than broad dictionary-based expansion for this task.

While LSTM-based sentence compression improved readability, it slightly decreased automatic ROUGE scores, highlighting the ongoing tension between human-centric readability and keyword-based evaluation metrics. Future work will focus on refining compression models and automatically tuning hyperparameters based on question types.

Summary updated

BibTeX
@inproceedings{chandu2017tackling,
  author = {Khyathi Raghavi Chandu and Aakanksha Naik and Aditya Chandrasekar and Zi Yang and Niloy Gupta and Eric Nyberg},
  title = {{Tackling Biomedical Text Summarization: OAQA at BioASQ 5B}},
  booktitle = {BioNLP},
  year = {2017},
  url = {https://openreview.net/forum?id=kkMi8-v7EfX},
}
Figure