Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension

Lin Ai, Zheng Hui, Zizhou Liu, Julia Hirschberg

Abstract

Machine Reading Comprehension (MRC) requires models to answer questions based strictly on a provided context. While generative models offer flexibility, they often suffer from factual hallucinations and ill-formed outputs. This paper introduces Question-Attended Span Extraction (QASE), a lightweight auxiliary module that grounds generative models by incorporating an extractive task during fine-tuning. Experimental results show that QASE-enhanced models, such as Flan-T5-Large, can outperform larger models like GPT-4 on benchmark datasets with negligible computational overhead.

Introduction

Traditional MRC has relied on extractive models (e.g., BERT) that identify specific spans within a text. While precise, these models lack the reasoning and synthesis capabilities of generative Large Language Models (LLMs) like T5 or Llama. However, generative models are prone to "out-of-control generation," producing answers that are either factually inconsistent with the source text or incomplete.

QASE bridges this gap by providing generative models with the precision of extractive methods. By teaching the model where to look during training, it improves internal representations of the context without sacrificing generative flexibility.

Methodology: The QASE Framework

The QASE module is an auxiliary component integrated into the model during the fine-tuning phase. It employs three primary mechanisms to ensure grounding:

  1. Multi-Head Attention (MHA): The module uses a dedicated attention mechanism where the question serves as the "query" and the context serves as the "key" and "value." This forces the model to align the question's intent with specific factual anchors in the text.
  2. IO Tagging: Span extraction is framed as a sequence tagging task using the "Inside-Outside" (IO) schema, classifying each context token as part of an answer span (I) or not (O).
  3. Multi-Task Learning: The model optimizes a joint loss function consisting of the standard Language Modeling (LM) loss for generation and the QASE sequence tagging loss.
The QASE architecture integrates a sequence tagging task with standard language modeling to improve grounding.

Crucially, the QASE module is only utilized during training. During inference, the module is discarded, resulting in zero additional computational cost at runtime.

Experimental Setup

The researchers evaluated QASE across three diverse extractive MRC benchmarks:

  • SQuAD v1.1: Single-span extractive QA.
  • MultiSpanQA: Multi-span, non-contiguous answer extraction.
  • Quoref: Coreferential reasoning tasks.

Results and Analysis

QASE-enhanced models demonstrated significant performance gains over vanilla fine-tuned baselines. Notably, Flan-T5-Large_QASE achieved an F1 score of 91.7% on SQuAD, surpassing two-shot GPT-4 (88.2%) and human performance (91.2%).

Performance comparison across SQuAD, MultiSpanQA, and Quoref datasets for various model sizes with and without QASE.

Key Findings

  • Efficiency: For the Flan-T5-Large model, QASE added only approximately 0.2% more trainable parameters, proving that architectural grounding does not require massive scaling.
  • Factual Consistency: Using the $Q^2$ metric, QASE-enhanced models consistently outperformed baselines, indicating a marked reduction in hallucinations.
  • Multi-Span Proficiency: On MultiSpanQA, the model outperformed leading leaderboard entries in Overlap F1, demonstrating its ability to handle complex, fragmented information.

Conclusion

QASE demonstrates that the "hallucination" problem in generative MRC is often a training objective issue rather than an inherent flaw in generative architectures. By aligning the question with the context through an auxiliary extractive task, models become significantly more reliable. Future work will explore scaling QASE to 70B+ parameter models and investigating semi-supervised methods to reduce dependency on gold span annotations.

Summary updated

Abstract

Machine Reading Comprehension (MRC) poses a significant challenge in the field of Natural Language Processing (NLP). While mainstream MRC methods predominantly leverage extractive strategies using encoder-only models such as BERT, generative approaches face the issue of out-of-control generation -- a critical problem where answers generated are often incorrect, irrelevant, or unfaithful to the source text. To address these limitations in generative models for MRC, we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning phase of pre-trained generative language models (PLMs), QASE significantly enhances their performance, allowing them to surpass the extractive capabilities of advanced Large Language Models (LLMs) such as GPT-4 in few-shot settings. Notably, these gains in performance do not come with an increase in computational demands. The efficacy of the QASE module has been rigorously tested across various datasets, consistently achieving or even surpassing state-of-the-art (SOTA) results, thereby bridging the gap between generative and extractive models in extractive MRC tasks.
BibTeX
@inproceedings{ai2024enhancing,
  author = {Lin Ai and Zheng Hui and Zizhou Liu and Julia Hirschberg},
  title = {{Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.560},
}
Figure