DUT-NLP at MEDIQA 2019: An Adversarial Multi-Task Network to Jointly Model Recognizing Question Entailment and Question Answering

Huiwei Zhou, Xuefei Li, Weihong Yao, Chengkun Lang, Shixian NingACL 2019

TL;DR

This paper introduces an Adversarial Multi-Task Network (AMTN) that jointly tackles two crucial medical NLP tasks: Recognizing Question Entailment (RQE) and Question Answering (QA). By leveraging a shared encoder with an "Interactive Transformer" and employing adversarial training, AMTN learns more robust and generalizable representations, leading to significant performance improvements over single-task models.

What Problem Does This Paper Address?

The increasing volume of online health queries presents a challenge for both users seeking accurate information and systems designed to provide it. Manually answering these queries is time-consuming, highlighting the need for effective automated Question Answering (QA) systems. However, accurately answering medical questions often requires understanding the nuances of how questions relate to each other, specifically whether one question's answer would also address another (Recognizing Question Entailment - RQE). Existing approaches often tackle RQE and QA in isolation, missing out on the potential benefits of their inherent relationship. The BioNLP 2019 shared tasks aimed to address these challenges by providing datasets for RQE and medical QA.

Methods

The core of the proposed solution is the Adversarial Multi-Task Network (AMTN). It's built upon a shared encoder that learns representations beneficial for both RQE and QA tasks.

  • BioBERT Embedding Layer: The model starts by using BioBERT, a pre-trained language model specifically for biomedical text, to generate rich word embeddings.
  • Interactive Transformer Layer: This is a novel component designed to capture long-range dependencies and model interactions between sentence pairs. Unlike standard Transformers where Q, K, and V come from the same input, the Interactive Transformer uses different sentences as inputs for Q and K/V, enabling a more dynamic interaction.
  • Combination Layer: After the Interactive Transformer, max pooling is used to obtain sentence representations, which are then combined with the [CLS] token representation to form a final sentence pair representation.
  • Task-Specific Classifiers: Separate classifiers are used for RQE and QA tasks, taking the shared representation as input.
  • Adversarial Training: To ensure the shared representations are truly general and not biased towards one task, an adversarial training strategy is employed. A task discriminator is introduced to try and predict which task (RQE or QA) a given representation belongs to. The shared encoder is trained to "fool" this discriminator, forcing it to learn representations that are task-invariant. This process aims to separate private task-specific features from the shared, generalizable ones.

Datasets

The experiments were conducted on datasets from the BioNLP 2019 RQE and QA shared tasks.

  • RQE Dataset: Contains 9120 question pairs (8588 for training, 302 for validation, 230 for test).
  • QA Dataset: Contains 3042 question-answer pairs (1701 for training, 234 for validation, 1107 for test).

The paper notes that the QA training data was oversampled three times to balance it with the RQE dataset.

Key Insights and Ideas

  • Synergy between RQE and QA: The paper strongly suggests that RQE and QA tasks are not independent. Understanding entailment relationships between questions can significantly aid in answering them, and vice-versa.
  • The Power of Interactive Transformers: The "Interactive Transformer" architecture proves crucial. By allowing sentences to query each other's representations, it captures richer inter-sentence semantic information than standard Transformers, which treat sentences more independently.
  • Adversarial Training for Purity: The adversarial training component is key to achieving "purer" shared representations. By forcing the model to confuse a task discriminator, it learns features that are genuinely common to both tasks, rather than inadvertently picking up task-specific biases.
  • BioBERT's Domain Advantage: Using BioBERT, a domain-specific pre-trained model, provides a significant boost over general-purpose models like BERT, especially for biomedical text.

Key Results

  • Overall Improvement: AMTN achieved significant improvements over single-task models on both RQE and QA tasks.
  • Multi-Task vs. Single-Task: Multi-task learning (AMTN-Discriminator) outperformed single-task networks (STN) by a considerable margin, demonstrating the benefit of shared representations. For instance, STN (QA+RQE) achieved 61.7% RQE and 71.4% QA accuracy, while AMTN-Discriminator reached 63.6% RQE and 74.5% QA.
  • Adversarial Multi-Task's Edge: The full AMTN model (with adversarial training) further improved upon the multi-task baseline. AMTN achieved 64.3% RQE and 75.8% QA accuracy, outperforming AMTN-Discriminator by 0.7% and 1.3% respectively.
  • Shared Encoder Performance: The Interactive Transformer, when combined with BioBERT embeddings, yielded the best results among the tested shared encoders (CNN, Bi-LSTM, Transformer), achieving 64.3% RQE and 75.8% QA accuracy.

Limitations and Future Work

  • Acronyms: The model struggles with acronyms, a common issue in biomedical text. Determining if two sentences refer to the same concept when one uses an acronym (e.g., "GIST" vs. "Gastrointestinal Stromal Tumor") is difficult without external knowledge. The authors suggest integrating prior knowledge as a future direction.
  • Ambiguous Samples: Some samples are inherently fuzzy or have labels that are difficult to interpret, leading to misclassifications.
  • Semantic Confusion: Complex syntax or phrase collocations can lead to semantic misunderstandings, where the model might recognize keywords but fail to grasp the nuanced relationship (e.g., identifying contradiction despite keyword overlap).
  • Private Encoder Performance: Introducing private encoders for each task (AMTN+Private Encoder) surprisingly reduced performance, suggesting that the overhead of private parameters might lead to overfitting on smaller datasets. The authors believe this highlights the effectiveness of their adversarial approach in extracting generalizable shared features.

Summary updated

BibTeX
@inproceedings{zhou2019dut,
  author = {Huiwei Zhou and Xuefei Li and Weihong Yao and Chengkun Lang and Shixian Ning},
  title = {{DUT-NLP at MEDIQA 2019: An Adversarial Multi-Task Network to Jointly Model Recognizing Question Entailment and Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure