Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction

Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang

Introduction

Generative Question Answering (QA) models have gained popularity due to their ability to synthesize information and provide natural-sounding responses. However, these models frequently suffer from "semantic drift"—a phenomenon where the generated answer deviates from the factual content of the source text. This can manifest as summarizing irrelevant sections or, more critically, generating answers that directly contradict the provided evidence (e.g., answering "Yes" when the text implies "No").

Example of semantic drift where the baseline model (PALM) generates an incorrect affirmative answer despite the passage's content.

As shown in the example above, standard generative models like PALM can be misled by context, failing to capture the core factual nuance required for an accurate response. The blue text in the passage highlights the rationale that the model should have prioritized.

The REAG Framework

To address this, researchers introduced the Rationale-Enriched Answer Generator (REAG). REAG is designed to "anchor" the generation process by forcing the model to identify the most relevant evidence before formulating an answer.

The framework consists of three primary components:

  1. Rationale Span Extraction: The encoder is tasked with an auxiliary objective: predicting which tokens in the passage constitute the "rationale"—the specific span most relevant to the question.
  2. Rationale-Aware Decoder: The decoder utilizes a cross-attention mechanism that prioritizes these rationale-aware hidden states. This ensures that the final natural language generation is grounded in the extracted facts.
  3. Joint Training with Linear Decay: The model is trained simultaneously on extraction and generation. A linear decay schedule is applied to the extraction loss, allowing the model to focus on identifying the rationale early in training before shifting focus to refining the linguistic quality of the answer.

Experimental Evaluation

The model was evaluated using the MS MARCO dataset, a large-scale benchmark for Machine Reading Comprehension that requires synthesizing answers from complex passages. The training set comprised 150,000 QA pairs, with evaluation performed on a development set of 12,000 pairs and a test set of 110,000 questions.

Results and Analysis

REAG demonstrated superior performance over existing state-of-the-art generative models.

Performance comparison of REAG against state-of-the-art models on the MS MARCO dataset.

As illustrated in the table above, REAG achieved a 70.98 ROUGE-L and 69.12 BLEU-1 score, outperforming models such as Masque and the baseline PALM.

Beyond automated metrics, the authors emphasized the importance of Semantic Accuracy. While ROUGE and BLEU measure word overlap, they often fail to capture whether the meaning of an answer is correct. Human evaluation on a subset of samples revealed that REAG improved semantic accuracy to 84.33%, compared to 81.67% for the baseline. Ablation studies further confirmed that the rationale extraction task is the critical driver of this performance gain; removing it led to a substantial drop in both automated and semantic scores.

Conclusion

REAG offers a robust solution to the problem of semantic drift in generative QA by integrating an extractive "highlighter" into the generative process. While currently limited to single-span rationales, this "extract-then-generate" approach provides a blueprint for more factually consistent and reliable conversational AI. Future work may extend this framework to multi-hop reasoning across disparate parts of a document.

Summary updated

BibTeX
@inproceedings{li2021addressing,
  author = {Chenliang Li and Bin Bi and Ming Yan and Wei Wang and Songfang Huang},
  title = {{Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.118},
}
Figure