QLSC: A Query Latent Semantic Calibrator for Robust Extractive Question Answering

Sheng Ouyang, Jianzong Wang, Yong Zhang, Zhitao Li, Ziqi Liang, Xulong Zhang, Ning Cheng, Jing Xiao

Introduction

Extractive Question Answering (EQA) is a core task in Machine Reading Comprehension (MRC), where a model must identify the specific span of text in a passage that answers a given query. While modern Transformer-based models like BERT and RoBERTa perform exceptionally well on standard benchmarks, they remain notoriously brittle when faced with linguistic variations. The Query Latent Semantic Calibrator (QLSC) is a novel "plug-in" module designed to address these vulnerabilities, ensuring that models provide consistent and accurate answers by mapping queries to a learned "semantic center."

The Robustness Challenge

The authors identify two critical weaknesses in current EQA models:

  1. Over-sensitivity: Models often produce different (and frequently incorrect) answers when a question is paraphrased (e.g., "Who are carnations suitable for?" vs. "Who can you give carnations to?").
  2. Over-stability: Models may fail to distinguish between questions that are syntactically similar but semantically distinct (e.g., asking about an event in the morning versus the afternoon).

While traditional fixes involve adversarial training—feeding the model millions of perturbed examples—this approach is computationally expensive and data-hungry. QLSC offers a more efficient architectural solution to calibrate the model’s understanding of query semantics without requiring massive additional datasets.

The QLSC Architecture

The QLSC module is an auxiliary component that can be integrated into various existing MRC frameworks. It operates through three primary stages:

1. Semantic Center Learning (SCL)

The system utilizes a randomly initialized "Information Matrix" to learn subspace semantic center features. This effectively creates high-dimensional "prototypes" representing different types of query intents.

2. Soft Semantic Feature Selection (SSFS)

Recognizing that not all learned features are equally relevant, this stage employs an attention mechanism to assign higher weights to semantically robust features while filtering out noise. The result is a "global latent semantic center"—a distilled representation of the query's core meaning.

3. Query Semantic Calibration (QSC)

This latent semantic center is fused back into the original query and passage embeddings via an attention mechanism. By anchoring the query to a stable semantic center, the model becomes less distracted by specific word choices or syntactic structures, focusing instead on the underlying intent.

Experimental Results

The researchers evaluated QLSC across two major benchmarks: DuReader_robust (a Chinese dataset for testing robustness) and SQuAD 1.1 (the industry-standard English EQA dataset).

Performance Gains

QLSC provided consistent improvements across all tested models:

  • Robustness: On the DuReader_robust dataset, adding QLSC to a BERT baseline improved the F1 score on the "Over-Sensitivity" test set by 1.58 points and the "Over-Stability" set by 5.77 points.
  • State-of-the-Art Accuracy: On SQuAD 1.1, RoBERTa equipped with QLSC achieved an Exact Match (EM) of 93.1 and an F1 of 96.8, outperforming standard RoBERTa and other competitive models like XLNet.
  • Efficiency: Training loss curves indicate that models with QLSC converge faster than vanilla counterparts, suggesting the module helps the model identify the "right" features more quickly.

Key Insights and Future Directions

The primary advantage of QLSC is its "plug-in" nature; it can be added to various backbones (BERT, RoBERTa, MacBERT, etc.) to boost performance without retraining from scratch. Visualizations using Principal Component Analysis (PCA) confirm that with QLSC, embeddings for paraphrased questions cluster more tightly than in standard models, providing a buffer against linguistic noise.

However, the module's performance is sensitive to the number of information features ($K$), with $K=32$ found to be optimal. While this study focused on extractive QA, the authors suggest that this semantic calibration approach could be extended to other NLP tasks and potentially generative models, serving as a general-purpose robustness booster for transformer-based architectures.

Summary updated

BibTeX
@inproceedings{ouyang2024qlsc,
  author = {Sheng Ouyang and Jianzong Wang and Yong Zhang and Zhitao Li and Ziqi Liang and Xulong Zhang and Ning Cheng and Jing Xiao},
  title = {{QLSC: A Query Latent Semantic Calibrator for Robust Extractive Question Answering}},
  booktitle = {IJCNN},
  year = {2024},
  doi = {10.1109/ijcnn60899.2024.10651462},
}
Figure