KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering

Jianing Wang, Chengyu Wang, Minghui Qiu, Qiuhui Shi, Hongbin Wang, Jun Huang, Ming Gao

Abstract

Extractive Question Answering (EQA) typically relies on fine-tuning pre-trained language models (PLMs) to predict the start and end indices of an answer span. However, this approach suffers from an objective gap between pre-training and fine-tuning, as well as semantic confusion between similar candidate spans, particularly in few-shot scenarios. This paper introduces Knowledge Enhanced Contrastive Prompting (KECP), a framework that reformulates EQA as a non-autoregressive generation task. By integrating external knowledge and employing span-level contrastive learning, KECP significantly improves performance in data-scarce environments, achieving a 75.45% F1 score on SQuAD 2.0 with only 16 training examples.

Introduction

Extractive Question Answering (EQA) is a fundamental task in machine reading comprehension where a model identifies the specific span within a passage that answers a given query. While standard approaches involving "pointer heads" on models like BERT or RoBERTa have been successful in high-resource settings, they struggle when training data is limited.

The authors identify two primary challenges in few-shot EQA:

  1. The Objective Gap: PLMs are pre-trained using Masked Language Modeling (MLM) to predict tokens, but are fine-tuned to predict numerical indices. This discrepancy hinders the efficient transfer of pre-trained knowledge.
  2. Semantic Confusion: In complex passages, multiple spans may appear semantically plausible. Without sufficient data, models often fail to distinguish the correct answer from nearby distractors.

The KECP Framework

KECP addresses these challenges by treating EQA as a prompt-based generation problem rather than a coordinate-pointing task. The framework consists of four main components:

1. Query Prompt Construction

Instead of a standard question-passage input, KECP transforms the question into a template using heuristic rules. For example, the question "What was one of the Norman's major exports?" is converted into a prompt like [MASK] [MASK] [MASK] was one of the Norman's major exports. This aligns the fine-tuning task with the model's original MLM objective.

2. Knowledge-aware Prompt Encoder (KPE)

To provide the model with deeper semantic context, KECP incorporates external information from the WikiData5M knowledge base through two mechanisms:

  • Passage Knowledge Injection (PKI): This enriches passage token embeddings with relevant entity information from the knowledge base.
  • Passage-to-Prompt Injection (PPI): A cross-attention mechanism allows the knowledge-enhanced passage information to flow directly into the [MASK] tokens of the query prompt, aiding the model in "filling in the blanks."

3. Span-level Contrastive Learning (SCL)

To mitigate semantic confusion, KECP employs contrastive learning. The model is trained to maximize the similarity between the query and the ground-truth answer span while simultaneously pushing away "negative spans." These negatives include distractors that are either semantically similar or physically adjacent to the correct answer, forcing the model to learn sharper decision boundaries.

4. Model-free Prefix-tree Decoder

To ensure the generated answer is always a valid substring of the original passage, KECP utilizes a trie (prefix-tree) based on the passage text during inference. This constrained decoding ensures the model remains "extractive" despite its generative prompting architecture.

Experimental Results

The authors evaluated KECP on SQuAD 2.0 and the MRQA 2019 Shared Task (comprising six diverse datasets: SQuAD 1.1, NewsQA, TriviaQA, SearchQA, HotpotQA, and Natural Questions).

Few-shot Performance

KECP demonstrated substantial gains in the 16-shot setting:

  • SQuAD 2.0: KECP achieved an 75.45% F1 score, significantly outperforming the previous state-of-the-art few-shot model, Splinter, which scored 53.05%.
  • MRQA Generalization: Across all six MRQA datasets, KECP consistently outperformed standard fine-tuning (RoBERTa/SpanBERT) and other prompt-tuning methods (such as P-tuning V2) by margins often exceeding 5-10% F1.

Ablation Studies

Ablation tests confirmed that both the Knowledge Injection (KPE) and Contrastive Learning (SCL) are critical. Removing either component resulted in a sharp decline in performance, illustrating that external semantic anchors and discriminative training are both necessary for high-performance few-shot QA.

Conclusion and Future Work

KECP demonstrates that bridging the gap between pre-training and fine-tuning through prompting, combined with external knowledge and contrastive learning, is a powerful strategy for few-shot EQA. While the current system relies on heuristic prompt construction, future work may explore automated prompt generation and the application of this framework to other tasks like multiple-choice or cloze-style reading comprehension. Additionally, the authors aim to investigate controllable text generation techniques to further refine answer production in open-ended settings.

Summary updated

BibTeX
@inproceedings{wang2022kecp,
  author = {Jianing Wang and Chengyu Wang and Minghui Qiu and Qiuhui Shi and Hongbin Wang and Jun Huang and Ming Gao},
  title = {{KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.206},
}
Figure