Introduction
Standard machine learning models for Reading Comprehension (RC) typically operate under the assumption that data points are independent and identically distributed (i.i.d.). In practice, this means a model treats related questions—such as "Which city is larger, X or Y?" and "Which city is smaller, X or Y?"—as entirely separate tasks. This independence often leads to logical inconsistency; a model might correctly identify the larger city but then name that same city when asked which is smaller. On datasets like ROPES, which emphasize these subtle linguistic pivots, standard models like RoBERTa frequently fail to perform better than random guessing.
To address this, we introduce Instance Bundles: groups of closely related question-answer pairs that differ by only a few words. By training on these bundles using Contrastive Estimation (CE), models are forced to learn the underlying logic that distinguishes similar queries, leading to significant improvements in both accuracy and consistency.
Methods: Contrastive Estimation on Bundles
While standard Maximum Likelihood Estimation (MLE) training requires a model to identify the correct answer from all possible text spans, Contrastive Estimation (CE) focuses the model's attention on a specific "neighborhood" of confusingly similar examples. We developed several loss functions to navigate these neighborhoods by partitioning the probability space in different ways.
As shown in the figure above, these partitions include:
- Answer Conditional: Fixing the question and distinguishing the correct answer from other plausible answers in the bundle.
- Question Conditional: Fixing the answer and distinguishing the correct question from others in the bundle.
- Joint/Full Partition: Simultaneously normalizing over all combinations of questions and answers within the bundle.
To create these bundles where they do not naturally occur, we employ three heuristics: Question Mining (finding similar existing questions via lexical overlap), Question Generation (automatically creating counter-questions by swapping superlatives or negating verbs), and Diverse Top-k Sampling (using a pre-trained model to generate likely but incorrect alternative answers).
Key Insights and Results
Our evaluation across HotpotQA (Comparison subset) and ROPES revealed that the Question Conditional loss often outperformed the Answer Conditional loss. This suggests that requiring a model to identify exactly which question a specific answer belongs to provides a superior supervision signal for reasoning.
As illustrated in the violin plot, CE acts as a powerful regularizer. While MLE often blurs the distinction between similar inputs, CE (especially when interpolated with MLE) forces the model to "zoom in" on the pivot words. This results in a much higher probability mass being assigned to the correct answer relative to its contrastive neighbor ($p(a|q)$ vs $p(a|\tilde{q})$).
Performance Gains
- ROPES: T5-large performance jumped from 65.7% to 76.6% EM (Exact Match) using Question Conditional CE.
- HotpotQA: Accuracy on comparison questions improved from 81.8% to 88.3% F1.
- Consistency: On HotpotQA, the consistency metric (correctly answering all questions in a bundle) nearly doubled, rising from 51.2% to 87.3%.
- Joint Inference: By allowing the model to see related questions simultaneously at test time, performance on HotpotQA reached 90.1 F1.
Conclusion and Future Work
Training with instance bundles significantly enhances a model's ability to handle subtle linguistic variations. The primary limitation remains the dependency on high-quality bundles; heuristics like Top-k sampling showed negligible gains on datasets like Quoref where the model was already overconfident. Future research will focus on more robust, automated methods for generating logical perturbations to create instance bundles for a wider variety of reasoning tasks.
Summary updated