AUEB at BioASQ 6: Document and Snippet Retrieval

George Brokos, Polyvios Liosis, Ryan McDonald, Dimitris Pappas, Ion Androutsopoulos

TL;DR

This paper presents AUEB's approach to the BioASQ 6 challenge, focusing on document and snippet retrieval in the biomedical domain. They demonstrate the effectiveness of end-to-end deep learning models, achieving top performance by extending existing architectures like PACRR and DRMM with novel context-aware mechanisms and attention-based features.

What Problem Does This Paper Address?

The BioASQ challenge aims to advance information retrieval (IR) and question answering (QA) in the biomedical domain. Specifically, this paper tackles two core tasks: document retrieval (finding relevant documents for a given biomedical question) and snippet retrieval (identifying the most relevant sentences or short passages within those documents). Historically, traditional IR techniques dominated these challenges. However, the authors aim to showcase the power of modern deep learning models for these complex tasks, demonstrating their ability to outperform traditional methods in a specialized domain.

Methods

The AUEB team employed deep learning models for both document and snippet retrieval, with a focus on end-to-end trainable architectures.

Document Retrieval: * PACRR-based Models: They extended the Position-Aware Convolutional Recurrent Relevance (PACRR) model. A key innovation was TERM-PACRR, which applied an MLP independently to each query-term encoding, aggregating the scores linearly. This approach, with fewer parameters, proved less prone to overfitting and performed better than the original PACRR. * DRMM-based Models: They adapted the Deep Relevance Matching Model (DRMM). The main contribution here is ABEL-DRMM (Attention-Based Element-wise DRMM). This model addresses DRMM's limitations by: * Introducing context-sensitive term encodings using a single dense layer (akin to a one-layer Temporal Convolutional Network) to capture the meaning of terms based on their neighbors. * Replacing DRMM's non-differentiable histogram-based encodings with an attention mechanism. This mechanism computes document-aware query-term encodings by attending to relevant document terms, allowing for end-to-end training. * They also explored extensions to ABEL-DRMM: a density-based extension that rewards dense matches within a document and a confidence extension that filters results based on a confidence score.

Snippet Retrieval: * They utilized the Basic CNN (BCNN) model, a component of the broader ABCNN architecture. BCNN takes a query and a document snippet (sentence) as input and learns to score their relevance. * A crucial post-processing step for snippet retrieval involved re-ranking snippets based on the relevance scores of their parent documents. This was important because the snippet model was trained assuming only relevant documents were provided.

Datasets

  • MEDLINE/PubMed Baseline 2018 collection: This formed the primary document collection, comprising approximately 18 million articles (titles and abstracts). Articles with only titles were excluded.
  • Training Data: Years 1-5 of the BioASQ data were used for training and tuning the models.
  • Evaluation Data: Year 6 of the BioASQ data (batches 1-5) was used for reporting test results.

Key Insights and Ideas

  • End-to-End Deep Learning Dominance: The paper strongly advocates for end-to-end deep learning models in biomedical IR, showing they can outperform traditional methods that previously dominated BioASQ.
  • Context is King: Incorporating context into term embeddings (as in ABEL-DRMM) significantly improves relevance scoring by allowing models to understand terms within their surrounding text.
  • Attention for Differentiability and Relevance: The attention mechanism in ABEL-DRMM not only makes the model end-to-end trainable but also effectively highlights relevant document terms for a given query term, leading to better document-aware encodings.
  • Post-processing Matters: Simple post-processing steps, like re-ranking snippets based on document scores, can significantly boost performance, especially when the snippet model is trained under specific assumptions about document relevance.
  • High Precision vs. Recall Trade-off: The high-precision snippet retrieval system (AUEB-NLP-5) achieved excellent results but highlighted a potential issue: feeding it only high-precision documents might cause it to miss relevant snippets from documents that were borderline relevant but not selected.

Key Results

  • AUEB's systems consistently scored at the top or near the top for all batches of the BioASQ 6 challenge.
  • For document retrieval, the models (PACRR- and DRMM-based) performed very closely, with less than 1 MAP point separating them. The experimental ABEL-DRMM (AUEB-NLP-4) generally scored best among AUEB submissions.
  • The high-precision system (AUEB-NLP-5) was the best for snippet selection, demonstrating the benefit of feeding it only highly relevant documents.
  • Overall, AUEB's submissions were competitive and, in aggregate, achieved top scoring positions, marking a shift towards deep learning in this domain.

Limitations and Future Work

  • Computational Expense: Deep learning models for document retrieval are computationally expensive, necessitating a pre-retrieval step (like BM25) to narrow down the search space.
  • Snippet Model Assumptions: The snippet retrieval model was trained assuming only relevant documents were provided. This can lead to issues if irrelevant documents are included in the input, potentially causing the model to rank snippets from irrelevant documents highly.
  • Future Work: The authors anticipate further gains from deep learning models as they improve and more training data becomes available. They also suggest that future systems will likely see even greater improvements from these advanced techniques.

Summary updated

Abstract

We present AUEB's submissions to the BioASQ 6 document and snippet retrieval tasks (parts of Task 6b, Phase A). Our models use novel extensions to deep learning architectures that operate solely over the text of the query and candidate document/snippets. Our systems scored at the top or near the top for all batches of the challenge, highlighting the effectiveness of deep learning for these tasks.
BibTeX
@inproceedings{brokos2018aueb,
  author = {George Brokos and Polyvios Liosis and Ryan McDonald and Dimitris Pappas and Ion Androutsopoulos},
  title = {{AUEB at BioASQ 6: Document and Snippet Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure