Learning with Instance Bundles for Reading Comprehension

Dheeru Dua, Pradeep Dasigi, Sameer Singh, Matt Gardner

Introduction

Standard machine learning models for Reading Comprehension (RC) typically operate under the assumption that data points are independent and identically distributed (i.i.d.). In practice, this means a model treats related questions—such as "Which city is larger, X or Y?" and "Which city is smaller, X or Y?"—as entirely separate tasks. This independence often leads to logical inconsistency; a model might correctly identify the larger city but then name that same city when asked which is smaller. On datasets like ROPES, which emphasize these subtle linguistic pivots, standard models like RoBERTa frequently fail to perform better than random guessing.

To address this, we introduce Instance Bundles: groups of closely related question-answer pairs that differ by only a few words. By training on these bundles using Contrastive Estimation (CE), models are forced to learn the underlying logic that distinguishes similar queries, leading to significant improvements in both accuracy and consistency.

Methods: Contrastive Estimation on Bundles

While standard Maximum Likelihood Estimation (MLE) training requires a model to identify the correct answer from all possible text spans, Contrastive Estimation (CE) focuses the model's attention on a specific "neighborhood" of confusingly similar examples. We developed several loss functions to navigate these neighborhoods by partitioning the probability space in different ways.

Visualizing different contrastive loss partitions used for training on instance bundles: (a) Answer Conditional, (b) Question Conditional, (c) Full partition, and (d) Joint.

As shown in the figure above, these partitions include:

  • Answer Conditional: Fixing the question and distinguishing the correct answer from other plausible answers in the bundle.
  • Question Conditional: Fixing the answer and distinguishing the correct question from others in the bundle.
  • Joint/Full Partition: Simultaneously normalizing over all combinations of questions and answers within the bundle.

To create these bundles where they do not naturally occur, we employ three heuristics: Question Mining (finding similar existing questions via lexical overlap), Question Generation (automatically creating counter-questions by swapping superlatives or negating verbs), and Diverse Top-k Sampling (using a pre-trained model to generate likely but incorrect alternative answers).

Key Insights and Results

Our evaluation across HotpotQA (Comparison subset) and ROPES revealed that the Question Conditional loss often outperformed the Answer Conditional loss. This suggests that requiring a model to identify exactly which question a specific answer belongs to provides a superior supervision signal for reasoning.

Distribution of probability mass for the correct answer given a specific question versus its contrastive neighbor across different training objectives.

As illustrated in the violin plot, CE acts as a powerful regularizer. While MLE often blurs the distinction between similar inputs, CE (especially when interpolated with MLE) forces the model to "zoom in" on the pivot words. This results in a much higher probability mass being assigned to the correct answer relative to its contrastive neighbor ($p(a|q)$ vs $p(a|\tilde{q})$).

Performance Gains

  • ROPES: T5-large performance jumped from 65.7% to 76.6% EM (Exact Match) using Question Conditional CE.
  • HotpotQA: Accuracy on comparison questions improved from 81.8% to 88.3% F1.
  • Consistency: On HotpotQA, the consistency metric (correctly answering all questions in a bundle) nearly doubled, rising from 51.2% to 87.3%.
  • Joint Inference: By allowing the model to see related questions simultaneously at test time, performance on HotpotQA reached 90.1 F1.

Conclusion and Future Work

Training with instance bundles significantly enhances a model's ability to handle subtle linguistic variations. The primary limitation remains the dependency on high-quality bundles; heuristics like Top-k sampling showed negligible gains on datasets like Quoref where the model was already overconfident. Future research will focus on more robust, automated methods for generating logical perturbations to create instance bundles for a wider variety of reasoning tasks.

Summary updated

Abstract

When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other. However, closely related questions and their corresponding answers are not independent, and leveraging these relationships could provide a strong supervision signal to a model. Drawing on ideas from contrastive estimation, we introduce several new supervision techniques that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding another cross entropy loss term that is used in addition to traditional maximum likelihood estimation. Our techniques require bundles of related question-answer pairs, which we can either mine from within existing data or create using various automated heuristics. We empirically demonstrate the effectiveness of training with instance bundles on two datasets -- HotpotQA and ROPES -- showing up to 11% absolute gains in accuracy.
BibTeX
@inproceedings{dua2021learning,
  author = {Dheeru Dua and Pradeep Dasigi and Sameer Singh and Matt Gardner},
  title = {{Learning with Instance Bundles for Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.584},
}
Figure