TL;DR
This paper introduces ContrastiveQA, a novel approach to extractive question answering (QA) that tackles the problem of models getting confused by similar-sounding but incorrect answers. By employing contrastive learning with carefully sampled positive and negative answer candidates, ContrastiveQA significantly improves accuracy on challenging QA benchmarks.
What Problem Does This Paper Address?
Large language models (LLMs) like ChatGPT have made impressive strides in natural language processing, but they still struggle with "hallucination," where they generate plausible but incorrect information. In the context of extractive Question Answering (QA), where the goal is to pinpoint the exact answer within a given text, this confusion is particularly problematic. Existing methods often focus on re-ranking potential answers, but they falter when multiple generated answers are very similar to the correct one, especially when distractors are present in the text. Humans can easily discern the correct answer even amidst similar options, but current QA models lack this nuanced comprehension. This paper aims to bridge this gap by improving a model's ability to distinguish correct answers from similar distractors.
Methods
The core of ContrastiveQA lies in its three-step training process:
- Candidate Answer Extraction: First, a pre-finetuned QA model (specifically, Longformer in their experiments) is used to extract multiple candidate answers for a given question and context. Instead of just taking the top-ranked answer, the model retrieves several candidates, ordered by confidence score.
- Positive and Negative Sampling: This is the novel part. For each question-answer pair in the training data, the system generates positive and negative samples from the extracted candidates. A positive sample is a candidate answer that exactly matches the ground truth answer in terms of its token span and content. A negative sample is any candidate answer that does not match the ground truth answer, even if it's semantically similar or shares some tokens. This process is crucial for teaching the model what to look for and what to avoid.
- ContrastiveQA with Contrastive Learning: The model is then trained using contrastive learning. This technique aims to pull semantically similar representations closer together in an embedding space and push dissimilar ones further apart. In this context, the model learns to associate the question with the positive answer candidate while pushing it away from the negative answer candidates. This is achieved by minimizing a combined loss function that includes a cross-entropy loss (LCE) and a contrastive loss (LQC). The contrastive loss specifically encourages the model to bring the question and positive samples closer while pushing negative samples away.
Datasets
The paper evaluates ContrastiveQA on four established extractive QA datasets:
- HotpotQA: A multi-hop QA dataset requiring reasoning, specifically using the "distractor" setting which includes misleading paragraphs.
- SQuAD (Stanford Question Answering Dataset): A widely used dataset where answers are spans of text within a given context.
- TriviaQA: A realistic text-based QA dataset with complex and varied questions.
- NewsQA: A reading comprehension dataset derived from CNN news articles, sometimes requiring reasoning.
Key Insights and Ideas
- The Power of Contrastive Sampling: The most significant insight is that simply re-ranking answers isn't enough when distractors are highly similar. By explicitly training the model to distinguish between correct and incorrect (but similar) answers using positive and negative sampling, a much more robust system can be built.
- Beyond Simple Similarity: The paper highlights that while cosine similarity can be a basic measure, using more sophisticated sentence embeddings (like those from Sentence-BERT) for the contrastive loss leads to better performance, suggesting that capturing deeper semantic relationships is key.
- Optimal Candidate Set Size: The experiments reveal that there's an optimal number of candidate answers to consider (around 15 in their experiments). Too few candidates might miss the correct answer, while too many can overwhelm the model and exceed computational limits, especially with long contexts.
Key Results
ContrastiveQA demonstrated superior performance across the evaluated datasets:
- On HotpotQA, ContrastiveQA achieved the best F1 score (83.9) among the tested models, outperforming strong baselines like Longformer (76.4) and CDLM (77.2). This is particularly notable given HotpotQA's challenging distractor setting.
- Similar improvements were observed on SQuAD, TriviaQA, and NewsQA, where ContrastiveQA consistently achieved the highest F1 scores (e.g., 93.8 on SQuAD, 81.7 on TriviaQA, and 77.8 on NewsQA).
- Ablation studies confirmed the effectiveness of the contrastive loss term, showing that its inclusion significantly boosted performance compared to using only cross-entropy loss.
Limitations and Future Work
The authors acknowledge several limitations:
- Computational Cost: Training Longformer, the backbone model used for long contexts, is computationally intensive, requiring significant time and GPU resources. This limits the ability to explore optimal hyperparameters and model weights exhaustively.
- Heuristic Candidate Selection: Determining the optimal number of candidate sets (k) was done through experimentation, suggesting a need for a more dynamic or adaptive approach.
For future work, they propose developing an adaptive mechanism to dynamically specify the number of samples required for contrastive learning, which could improve efficiency and performance. They also aim to address the computational resource constraints, potentially through model weight reduction techniques, to better learn optimal framework parameters.
Summary updated