Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering

Kosuke Akimoto, Kunihiro Takeoka, Masafumi Oyamada

Abstract

Fusion-in-Decoder (FiD) models for open-domain question answering (ODQA) overfit to the specific context quality—the ratio of relevant to irrelevant passages—encountered during training. This mismatch between training and deployment environments leads to significant performance degradation. We demonstrate that models develop distinct "reading strategies" (selective vs. uniform attention) based on training quality. We propose a training-free inference-time intervention using attention temperature to adapt models to varying retrieval environments.

Introduction

Retrieval-Augmented Generation (RAG) models like Fusion-in-Decoder (FiD) are central to modern open-domain question answering (ODQA). These models retrieve documents and fuse them in a decoder to generate answers. While the benefits of context quantity and quality at inference time are well-documented, the impact of context quality during training remains less understood. In practice, retrieval systems are inconsistent; a model trained on high-quality data may face noisy contexts in production, leading to a "quality gap" that significantly degrades performance.

Methodology

We conducted a systematic study using the FiD architecture (T5-base) on the Natural Questions (NQ) and TriviaQA benchmarks. We controlled two training variables: context quantity ($n$) and context quality (the proportion of relevant passages). Relevant passages were strictly defined using a secondary filtering step with a pre-trained QA model.

Analysis: The "Selectivity" Trap

Our analysis of cross-attention distributions reveals that models develop different "reading strategies" based on training quality. Models trained on low-quality context become highly selective, attending sharply to specific passages (a "searchlight" pattern). Conversely, models trained on high-quality context attend more uniformly across documents (a "floodlight" pattern).

Cross-attention probability distributions for relevant and irrelevant passages across different training qualities. High-quality training leads to more uniform attention, while low-quality training results in highly selective attention.

When a "selective" model encounters high-quality context at inference time, it continues to ignore most relevant passages, effectively discarding useful information.

Experimental Results

The experiments revealed a phenomenon of "diagonal dominance": models consistently achieve peak performance when the evaluation environment matches the training environment's quality level.

Exact Match (EM) scores across different training and evaluation configurations. The bolded diagonal values highlight that performance is maximized when training and evaluation qualities align.

We found that while context quantity during training has a negligible effect, context quality is the primary driver of model behavior. To address the quality gap without retraining, we introduced Temperature-based Adaptation. By adjusting a temperature parameter ($T$) in the cross-attention mechanism during inference, we can manually sharpen or flatten the model's focus to match the current retrieval environment.

Conclusion and Future Work

Context quality is a critical factor in training robust FiD models. Our proposed temperature adaptation successfully recovers performance lost due to environment mismatch. Future work includes exploring this phenomenon in abstractive tasks and developing dynamic temperature mechanisms that adapt to individual queries in real-time.

Summary updated

BibTeX
@inproceedings{akimoto2023context,
  author = {Kosuke Akimoto and Kunihiro Takeoka and Masafumi Oyamada},
  title = {{Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.784},
}
Figure