Abstract
We present AUEB's submissions to the BioASQ 6 document and snippet retrieval tasks (parts of Task 6b, Phase A). Our models use novel extensions to deep learning architectures that operate solely over the text of the query and candidate document/snippets. Our systems scored at the top or near the top for all batches of the challenge, highlighting the effectiveness of deep learning for these tasks.
Introduction
In the biomedical domain, information retrieval (IR) is a critical task, given the vast and rapidly growing volume of literature—currently exceeding 28 million articles in PubMed. The BioASQ challenge provides a benchmark for evaluating systems on two primary tasks: Document Retrieval (identifying relevant MEDLINE articles) and Snippet Retrieval (extracting specific relevant sentences). Historically, statistical keyword-matching techniques like BM25 have dominated this space. However, recent advancements in neural IR suggest that deep learning can better capture the semantic nuances of complex biomedical language.
System Overview
Due to the computational intensity of deep neural networks, it is infeasible to score all 28 million documents for every query. Our system employs a multi-stage pipeline. First, a traditional BM25 engine performs a "pre-retrieval" step to narrow the candidate pool to the top $N$ documents. These candidates are then processed by our deep document ranking models. The top 10 documents from this stage are further analyzed to extract candidate snippets, which are then ranked by a dedicated snippet retrieval model.
Document Retrieval Models
We introduced two primary architectural extensions to improve document ranking performance:
TERM-PACRR
This is an extension of the Position-Aware Convolutional Recurrent Relevance (PACRR) model. Unlike the original architecture, TERM-PACRR uses a shared Multi-Layer Perceptron (MLP) to score each query term independently against the document. This approach reduces the number of parameters, mitigating the risk of overfitting on the relatively small BioASQ training sets.
ABEL-DRMM
The Attention-Based Element-wise DRMM (ABEL-DRMM) is our most advanced document retrieval model. It addresses two limitations of the standard Deep Relevance Matching Model (DRMM):
- Context Sensitivity: By utilizing a one-layer Temporal Convolutional Network (TCN), the model encodes terms based on their local neighbors, allowing it to distinguish between different semantic uses of the same word.
- Differentiability: We replaced non-differentiable histograms with an attention mechanism and Hadamard products. This allows the model to be trained end-to-end, learning a more nuanced interaction between query terms and document content.
Snippet Retrieval
For snippet retrieval, we utilized a Basic Bi-CNN (BCNN) architecture. This model processes the query and a candidate sentence in parallel through convolutional layers to determine relevance. A key finding in our work was the importance of post-processing: re-ranking snippets based on the relevance score of their parent document significantly improved the final precision of the system.
Experimental Setup and Results
Models were trained using data from BioASQ years 1–5 and evaluated on the BioASQ 6 (Task 6b, Phase A) benchmark. We utilized 200-dimensional word2vec embeddings pre-trained on the PubMed collection to ensure domain-specific semantic coverage.
Our systems consistently performed at or near the top of the leaderboard across all five batches of the challenge:
- Document Retrieval: ABEL-DRMM variants achieved a Mean Average Precision (MAP) of up to 0.1271 in Batch 1, outperforming the top non-AUEB competitor baseline of 0.1058.
- Snippet Retrieval: In Batch 5, our system achieved an F1 score of 0.0778, nearly 50% higher than the next best non-AUEB competitor (0.0542).
Conclusion
The success of AUEB's submissions at BioASQ 6 demonstrates that context-sensitive, term-based interaction models are highly effective for biomedical IR. While traditional features like BM25 remain useful for initial filtering, deep learning models that capture local context and term density provide a significant performance boost in identifying the most relevant documents and snippets.
Summary updated