Abstract
Fusion-in-Decoder (FiD) models, a standard for retrieval-augmented generation, suffer from overfitting to the "cleanliness" (ratio of relevant to irrelevant passages) of their training data. This study investigates the impact of context quantity and quality during training, revealing that context quality dictates the model's cross-attention strategy. Models trained on high-quality context adopt a uniform attention pattern, while those trained on low-quality context become highly selective. We demonstrate that this mismatch leads to significant performance degradation and propose a training-free temperature adjustment during inference to recover performance.
Introduction
Open-domain question answering (ODQA) typically follows a "retrieve-then-read" pipeline. A retriever pulls dozens of passages from a corpus, and a model like Fusion-in-Decoder (FiD) processes them to generate an answer. While the benefits of providing more or better passages during inference are well-documented, the influence of these factors during training is less understood. This research explores the "mismatch" problem: how context quantity (the absolute number of passages) and context quality (the proportion of relevant passages) during training affect a model's ability to generalize to different real-world retrieval environments.
Methods
The researchers conducted controlled experiments using the FiD architecture (based on T5-base) on the Natural Questions (NQ) and TriviaQA benchmarks. Their approach involved:
- Controlled Environments: Creating training and evaluation sets with strictly controlled numbers of relevant ($n^+$) and irrelevant ($n^-$) passages.
- Cross-Attention Analysis: Analyzing aggregated cross-attention scores to determine if the model was being "selective" (skeptical) or "uniform" (trusting) toward the provided context.
- Temperature Intervention: Introducing a temperature parameter ($T$) to the cross-attention distribution during inference. By adjusting $T$, they could manually sharpen or flatten the model's attention to simulate different training behaviors without retraining.
Key Insights: The Attention Signature
The study reveals that FiD models overfit heavily to context quality rather than quantity. This behavior is rooted in an "attention signature" developed during training. High-quality training environments encourage uniform attention, where the model learns to trust most provided passages. Conversely, low-quality environments lead to selective attention, where the model becomes skeptical and focuses intensely on only a few passages while ignoring the rest.
When a "selective" model (trained on low quality) is given high-quality context, it remains too picky, overlooking useful information. Conversely, a "uniform" model (trained on high quality) gets distracted by noise when context quality drops in the evaluation environment.
Results and Adaptation
A significant performance mismatch occurs when the evaluation environment's context quality differs from the training environment. As shown in the experimental results, models consistently perform best when the evaluation environment matches the training environment (indicated by the peaks in performance curves).
The proposed temperature adaptation effectively bridges this gap. By increasing the temperature for models trained on low-quality data, the authors forced a more uniform attention pattern, significantly improving performance on high-quality evaluation sets. This training-free method allows engineers to adapt pre-trained FiD models to new retrieval systems simply by tuning a single hyperparameter during inference.
Conclusion
This work highlights that context quality is a critical factor in training robust FiD models. While the study focused on extractive QA with T5-base, the findings suggest that matching training context quality to deployment conditions—or using inference-time adjustments—is essential for optimal performance. Future work may explore instance-specific adaptation, predicting the best temperature for each individual question based on the retrieved context's estimated quality.
Summary updated