On Committee Representations of Adversarial Learning Models for Question-Answer Ranking

Sparsh Gupta, Vitor CarvalhoACL 2019

TL;DR

This paper tackles the challenge of adversarial training in Question-Answer (QA) ranking, where models can sometimes degrade in performance due to overfitting. They propose a novel "adversarial committee learning" strategy that samples models at different stages of adversarial training to form a diverse committee, significantly improving robustness and achieving state-of-the-art results on multiple QA ranking benchmarks.

What Problem Does This Paper Address?

Question-Answer (QA) ranking aims to accurately order potential answers to a given question. While deep learning models have made significant strides, adversarial training, a technique to make models more robust by exposing them to intentionally deceptive inputs, can sometimes lead to performance degradation. This happens when models overfit to the adversarial examples, forgetting their knowledge of the original, clean data. The paper investigates this phenomenon and proposes a solution to harness the benefits of adversarial training without its drawbacks.

Methods

The core contribution is the Adversarial Committee Learning strategy. Instead of using a single, adversarially trained model, this approach creates a committee of models. Specifically, models are sampled at regular intervals during the adversarial training process, including the initial pre-trained model and the final adversarially trained model. These sampled models, each with decision boundaries tuned to different proportions of the original and adversarial datasets, form a diverse committee. During prediction, the scores from each committee member are combined, weighted by their performance on a validation set. This ensemble approach aims to mitigate overfitting by leveraging the collective knowledge of diverse models.

The paper also discusses two baseline model types:

  • Representation-based models: Map questions and answers to a shared representation space.
  • Interaction-based models: Capture multiple levels of similarity between questions and answers.

They also detail the Multihop Attention Network (MAN), a state-of-the-art representation-based model that uses sequential attention across multiple layers. For adversarial training, they adapt IRGAN (Information Retrieval Generative Adversarial Networks), which uses a generator to create adversarial examples and a discriminator to learn ranking on them, trained using reinforcement learning.

Datasets

The paper evaluates their approach on four datasets:

  • WikiQA: An open-domain QA dataset.
  • Insurance QA: A financial domain non-factoid QA dataset.
  • FIQA: A financial domain non-factoid QA dataset.
  • Tax Domain QA: A dataset focused on tax-related questions.

These datasets vary in size and domain, allowing for a comprehensive evaluation of the proposed method.

Key Insights and Ideas

  • Adversarial training isn't always beneficial: While often improving robustness, adversarial training can lead to overfitting and performance degradation, especially for models with higher capacity like MAN.
  • Overfitting is a key issue: Models can become too specialized in ranking adversarial examples, losing their ability to generalize to clean, unseen data.
  • Diversity is key to robustness: By combining models trained at different stages of adversarial training, the committee approach leverages diverse decision-making capabilities, effectively mitigating overfitting.
  • Committee learning as a regularization technique: The adversarial committee strategy acts as a form of regularization, preventing any single model from becoming overly specialized.

Key Results

The proposed Adversarial Committee (AdvCom) strategy consistently improved the performance of all baseline models tested, including Match Pyramid, Deep Matching Net, and MAN.

  • For MAN, vanilla adversarial training significantly worsened performance, but AdvCom-MAN achieved state-of-the-art results on almost all tested datasets.
  • AdvCom-Match Pyramid and AdvCom-Deep Matching Net also showed significant improvements over their vanilla adversarial counterparts.
  • The paper reports that AdvCom-MAN outperformed the previous state-of-the-art by as much as 6% in NDCG (Normalized Discounted Cumulative Gain) on some datasets.

Limitations and Future Work

  • Model Capacity and Overfitting: The paper hypothesizes that models with higher capacity (like MAN) are more prone to overfitting during adversarial training. This suggests that the effectiveness of adversarial training might be inversely related to model capacity without proper regularization.
  • Sampling Strategy: While the paper found sampling every 3rd epoch to be optimal, further exploration of different sampling strategies for committee formation could be beneficial.
  • Generalizability: While tested on four datasets, further validation on a wider range of QA tasks and domains would strengthen the claims.

The authors suggest that their adversarial committee learning strategy is a promising direction for improving the robustness and performance of QA ranking models, particularly in the presence of adversarial attacks or noisy data.

Summary updated

BibTeX
@inproceedings{gupta2019committee,
  author = {Sparsh Gupta and Vitor Carvalho},
  title = {{On Committee Representations of Adversarial Learning Models for Question-Answer Ranking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure