Abstract
Question Answering (QA) models based on pretrained language models have achieved remarkable performance on various benchmark datasets. However, these models often fail to generalize to unseen data outside the training distribution due to distributional shifts. While traditional Data Augmentation (DA) techniques—such as word dropping or replacement—can regularize models, they often introduce semantic changes that lead to incorrect answers. To address this, we propose Stochastic Word Embedding Perturbation (SWEP), a method based on a stochastic noise generator that learns to perturb word embeddings without altering their underlying semantics. Evaluated across five target domains using a single source dataset, SWEP significantly outperforms baseline DA methods and even surpasses models trained on over 240,000 artificially generated QA pairs.
Introduction
Deep learning models for Question Answering, such as BERT or T5, are notoriously brittle. While they excel in-distribution, their performance typically plummets when faced with out-of-distribution (OOD) text, such as shifting from Wikipedia articles to Reddit posts or medical journals.
The standard solution for improving robustness is Data Augmentation (DA). However, text-based DA is uniquely challenging. Unlike computer vision, where flipping an image of a cat still results in a cat, replacing a single word in a question (e.g., a proper name) can completely change the required answer or render the context nonsensical. This "semantic drift" is particularly damaging for QA tasks, where the precise relationship between the question and the context is critical.
Methodology: Stochastic Word Embedding Perturbation (SWEP)
SWEP moves away from manipulating discrete words and instead operates in the continuous embedding space. The core of the method is a stochastic noise generator designed to identify which parts of an input are semantically "load-bearing."
1. The Noise Generator
The system employs a small Multi-Layer Perceptron (MLP) situated atop a standard transformer encoder. This generator analyzes the contextualized representation of each word to predict a specific mean ($\mu$) and variance ($\sigma^2$) for noise.
2. Multiplicative Perturbation
Rather than additive noise, which can be easily ignored by high-magnitude embeddings, SWEP uses multiplicative perturbation. The predicted noise is multiplied by the original word embedding, effectively acting as an "input-dependent dropout" mechanism that scales its intensity based on the word's importance to the task.
3. Objective Function and Training
The noise generator is trained with a dual-objective function:
- Maximum Likelihood: The perturbed input must still allow the QA model to identify the correct answer, ensuring that the noise does not destroy the essential semantics.
- KL Divergence: The noise distribution is regularized against a prior distribution to ensure diversity and prevent the generator from collapsing into a trivial state.
To ensure stability, the authors utilize a stop-gradient trick. This prevents the main QA model from simply learning to bypass the noise during the early stages of training, forcing it to become truly robust to the perturbations.
Experimental Setup
The models were trained on a single source dataset, SQuAD (Wikipedia-based), and evaluated across five distinct OOD target domains:
- BioASQ: Biomedical literature.
- New York Times: News articles.
- Reddit: Social media posts.
- Amazon: Product reviews.
- New Wikipedia: A separate Wikipedia split to test temporal or topical shifts.
The method was validated using both extractive models (BERT-base) and generative models (T5-small).
Key Insights
Semantics Over Syntax
By perturbing embeddings rather than tokens, the model learns a "range of tolerance" for each word. Analysis shows that the generator perturbs answer-critical words (like proper names or specific dates) much less than filler words. This suggests the model successfully learns which linguistic elements are essential for maintaining the correct answer.
Frequency-Dependent Robustness
The research found that low-frequency words are perturbed more aggressively than high-frequency words. Because low-frequency words are more sparsely distributed in the embedding space, they can be moved further without overlapping with the meaning of another word, allowing for more diverse augmentation without semantic corruption.
Efficiency vs. Scale
A significant finding of this work is that SWEP—which adds fewer than 2 million parameters to a 110M parameter BERT model—outperformed a baseline model trained on over 240,000 artificially generated QA pairs. This indicates that high-quality, targeted perturbations are more effective for OOD generalization than massive amounts of synthetic text.
Results
SWEP consistently outperformed standard Maximum Likelihood Estimation (MLE) and other DA methods like Word-Dropout and Adversarial Augmentation across nearly all OOD datasets.
- BioASQ Performance: On the challenging medical dataset, SWEP achieved an F1 score of 61.72, compared to the baseline MLE's 59.77.
- Low-Resource Scenarios: When only 10% of the SQuAD training data was available, SWEP still improved Exact Match (EM) and F1 scores by approximately 1%, demonstrating its utility as a regularizer when data is scarce.
- Generative QA: The method proved equally effective for T5 models, showing that the benefits of embedding perturbation extend beyond extractive "span-pointing" to full text generation.
Conclusion
SWEP provides a simple yet effective framework for improving the robustness of QA models. By learning to inject noise that respects the semantic boundaries of the input, it provides a more reliable form of data augmentation than discrete word manipulation. Future work may explore the application of SWEP to other sensitive NLP tasks, such as summarization or formal reasoning, where maintaining semantic integrity is paramount.
Summary updated