Abstract
Generating "ideal answers" to biomedical questions requires synthesizing concise, accurate summaries from vast amounts of technical literature. Researchers from Carnegie Mellon University developed an extractive summarization system for the BioASQ challenge that leverages a novel word-embedding-based similarity metric and "soft positional constraints." Their pipeline achieved top-tier performance by effectively bridging the semantic gap between queries and source documents.
Introduction
The exponential growth of the MEDLINE database, which now contains over 23 million references, presents a significant challenge for medical researchers. While traditional search engines can retrieve relevant documents, the task of Ideal Answer Generation—creating a concise, non-redundant, and readable summary that directly answers a specific query—remains difficult. This paper describes the OAQA system's approach to Phase B of the BioASQ challenge, focusing on the semantic understanding required to link questions with answer sentences across multiple documents.
Methodology
The OAQA system employs a three-stage pipeline: Relevance Ranking, Sentence Selection, and Sentence Tiling.
Key Components:
- Granularity and Filtering: The team found that using pre-filtered "snippets" provided by the challenge yielded better results than extracting sentences from full PubMed abstracts, as snippets are already high-relevance candidates.
- w2v tf-idf Similarity: To move beyond exact keyword matching, the authors integrated Word2Vec embeddings (pre-trained on PubMed and Wikipedia) into a tf-idf framework. This allows the system to recognize semantic relationships between different terms, such as a drug and its therapeutic class.
- Soft Positional Constraints: Recognizing that the order of snippets often reflects their importance, a weighting parameter ($\alpha$) was introduced to boost the scores of sentences appearing earlier in the source list.
- Maximal Marginal Relevance (MMR): MMR was utilized to balance query relevance with information novelty, effectively reducing redundancy in the final summary.
- Sentence Compression: An LSTM-based neural network was tested to remove redundant introductory phrases (e.g., "We report that..."), improving readability.
Results and Evaluation
The system was evaluated on the BioASQ 4 and 5B datasets. The combination of MMR, soft positional constraints, and w2v tf-idf similarity proved to be the most effective configuration.
As shown in the results table, the "Soft positional constraint + w2v tf-idf similarity" configuration (Experiment 17) achieved the highest scores, with a ROUGE-2 of 0.6534 and a ROUGE-SU4 of 0.6536. This significantly outperformed the "first snippet" baseline (Experiment 12), which scored only 0.3363 in ROUGE-2.
Discussion
A key insight from the study was the value of the "position signal" in the source snippets. Simply favoring earlier content provided a measurable performance boost. Conversely, while medical ontologies like UMLS and SNOMEDCT are valuable, "unbounded" expansion often introduced noise that hindered performance. This suggests that controlled semantic similarity via embeddings is more robust than broad dictionary-based expansion for this task.
While LSTM-based sentence compression improved readability, it slightly decreased automatic ROUGE scores, highlighting the ongoing tension between human-centric readability and keyword-based evaluation metrics. Future work will focus on refining compression models and automatically tuning hyperparameters based on question types.
Summary updated