Abstract
Fusion-in-Decoder (FiD) models for open-domain question answering (ODQA) overfit to the specific context quality—the ratio of relevant to irrelevant passages—encountered during training. This mismatch between training and deployment environments leads to significant performance degradation. We demonstrate that models develop distinct "reading strategies" (selective vs. uniform attention) based on training quality. We propose a training-free inference-time intervention using attention temperature to adapt models to varying retrieval environments.
Introduction
Retrieval-Augmented Generation (RAG) models like Fusion-in-Decoder (FiD) are central to modern open-domain question answering (ODQA). These models retrieve documents and fuse them in a decoder to generate answers. While the benefits of context quantity and quality at inference time are well-documented, the impact of context quality during training remains less understood. In practice, retrieval systems are inconsistent; a model trained on high-quality data may face noisy contexts in production, leading to a "quality gap" that significantly degrades performance.
Methodology
We conducted a systematic study using the FiD architecture (T5-base) on the Natural Questions (NQ) and TriviaQA benchmarks. We controlled two training variables: context quantity ($n$) and context quality (the proportion of relevant passages). Relevant passages were strictly defined using a secondary filtering step with a pre-trained QA model.
Analysis: The "Selectivity" Trap
Our analysis of cross-attention distributions reveals that models develop different "reading strategies" based on training quality. Models trained on low-quality context become highly selective, attending sharply to specific passages (a "searchlight" pattern). Conversely, models trained on high-quality context attend more uniformly across documents (a "floodlight" pattern).
When a "selective" model encounters high-quality context at inference time, it continues to ignore most relevant passages, effectively discarding useful information.
Experimental Results
The experiments revealed a phenomenon of "diagonal dominance": models consistently achieve peak performance when the evaluation environment matches the training environment's quality level.
We found that while context quantity during training has a negligible effect, context quality is the primary driver of model behavior. To address the quality gap without retraining, we introduced Temperature-based Adaptation. By adjusting a temperature parameter ($T$) in the cross-attention mechanism during inference, we can manually sharpen or flatten the model's focus to match the current retrieval environment.
Conclusion and Future Work
Context quality is a critical factor in training robust FiD models. Our proposed temperature adaptation successfully recovers performance lost due to environment mismatch. Future work includes exploring this phenomenon in abstractive tasks and developing dynamic temperature mechanisms that adapt to individual queries in real-time.
Summary updated