Abstract
Machine Reading Comprehension (MRC) requires models to answer questions based strictly on a provided context. While generative models offer flexibility, they often suffer from factual hallucinations and ill-formed outputs. This paper introduces Question-Attended Span Extraction (QASE), a lightweight auxiliary module that grounds generative models by incorporating an extractive task during fine-tuning. Experimental results show that QASE-enhanced models, such as Flan-T5-Large, can outperform larger models like GPT-4 on benchmark datasets with negligible computational overhead.
Introduction
Traditional MRC has relied on extractive models (e.g., BERT) that identify specific spans within a text. While precise, these models lack the reasoning and synthesis capabilities of generative Large Language Models (LLMs) like T5 or Llama. However, generative models are prone to "out-of-control generation," producing answers that are either factually inconsistent with the source text or incomplete.
QASE bridges this gap by providing generative models with the precision of extractive methods. By teaching the model where to look during training, it improves internal representations of the context without sacrificing generative flexibility.
Methodology: The QASE Framework
The QASE module is an auxiliary component integrated into the model during the fine-tuning phase. It employs three primary mechanisms to ensure grounding:
- Multi-Head Attention (MHA): The module uses a dedicated attention mechanism where the question serves as the "query" and the context serves as the "key" and "value." This forces the model to align the question's intent with specific factual anchors in the text.
- IO Tagging: Span extraction is framed as a sequence tagging task using the "Inside-Outside" (IO) schema, classifying each context token as part of an answer span (I) or not (O).
- Multi-Task Learning: The model optimizes a joint loss function consisting of the standard Language Modeling (LM) loss for generation and the QASE sequence tagging loss.
Crucially, the QASE module is only utilized during training. During inference, the module is discarded, resulting in zero additional computational cost at runtime.
Experimental Setup
The researchers evaluated QASE across three diverse extractive MRC benchmarks:
- SQuAD v1.1: Single-span extractive QA.
- MultiSpanQA: Multi-span, non-contiguous answer extraction.
- Quoref: Coreferential reasoning tasks.
Results and Analysis
QASE-enhanced models demonstrated significant performance gains over vanilla fine-tuned baselines. Notably, Flan-T5-Large_QASE achieved an F1 score of 91.7% on SQuAD, surpassing two-shot GPT-4 (88.2%) and human performance (91.2%).
Key Findings
- Efficiency: For the Flan-T5-Large model, QASE added only approximately 0.2% more trainable parameters, proving that architectural grounding does not require massive scaling.
- Factual Consistency: Using the $Q^2$ metric, QASE-enhanced models consistently outperformed baselines, indicating a marked reduction in hallucinations.
- Multi-Span Proficiency: On MultiSpanQA, the model outperformed leading leaderboard entries in Overlap F1, demonstrating its ability to handle complex, fragmented information.
Conclusion
QASE demonstrates that the "hallucination" problem in generative MRC is often a training objective issue rather than an inherent flaw in generative architectures. By aligning the question with the context through an auxiliary extractive task, models become significantly more reliable. Future work will explore scaling QASE to 70B+ parameter models and investigating semi-supervised methods to reduce dependency on gold span annotations.
Summary updated