Abstract
Detecting negatives—such as unanswerable questions or false claims—is a critical yet challenging component of natural language understanding (NLU). While human-annotated negative examples are effective, they are expensive to produce and often contain domain-specific biases. We introduce SCENE (Self-labeled Counterfactuals for Extrapolating to Negative Examples), an automated framework that synthesizes challenging negative training data from purely positive examples. By perturbing positive instances and applying a self-training heuristic, SCENE enables models to extrapolate to negative classes zero-shot. Our method closes 69.6% of the performance gap on SQuAD 2.0 compared to models trained on human-labeled negatives and demonstrates strong generalization across boolean QA and textual entailment tasks.
Introduction: The Challenge of Negative Examples
Robust NLU requires models to distinguish between supported claims (positives) and unsupported ones (negatives). In extractive Question Answering (QA), for instance, a model must determine if a question is answerable based on the provided context.
High-quality negative examples are difficult to acquire. Human-annotated datasets like SQuAD 2.0 are costly and often suffer from "annotation artifacts"—simple patterns that models exploit without true comprehension. Conversely, automated methods like "distant supervision" (pairing random questions with unrelated text) create negatives that are too simplistic. SCENE addresses this by generating "subtle" negatives that force models to learn fine-grained semantic distinctions.
Methodology: The SCENE Pipeline
SCENE generates challenging negatives on-the-fly during training through a three-step process:
- Perturb: Starting with a positive example, SCENE masks tokens in the question and uses a BART model to infill them. This creates a "counterfactual" question that remains linguistically similar to the original but may no longer be answerable.
- Filter: To ensure quality, two filters are applied:
- Paraphrase Detector: A RoBERTa-based model ensures the new question isn't a mere paraphrase of the original, which would likely retain the same answer.
- Model Consistency: The current QA model's prediction for the perturbed question is compared to the original. If the perturbation fails to change the predicted answer or causes extreme confusion, the example is discarded.
- Self-Label: Examples passing these filters are labeled as "unanswerable" (or "not-entailed") and integrated into the training set. This self-training curriculum progressively challenges the model with increasingly difficult negatives.
Key Insights: The Power of Subtle Perturbations
The effectiveness of SCENE lies in its ability to create "subtle" negatives—examples that look nearly identical to positives but contain a small semantic shift (e.g., swapping an entity or changing a verb) that flips the label.
Surprisingly, models trained with SCENE on SQuAD 1.1 (positives only) often generalize better to out-of-domain data (like ACE-whQA) than models trained on human-annotated SQuAD 2.0. This suggests that synthetic counterfactuals can be more diverse and less prone to the biases found in human-written negatives.
Experimental Results
The researchers evaluated SCENE across several benchmarks:
- Extractive QA: On SQuAD 2.0, SCENE closed 69.6% of the performance gap between a model trained only on positives and one trained on the full human-annotated dataset.
- Boolean QA: On BoolQ-3L, the method closed 89.3% of the gap, effectively teaching the model to handle "Yes/No/IDK" scenarios.
- Textual Entailment: For RTE, SCENE closed 56.1% of the gap using only entailment examples as a starting point.
These results demonstrate a powerful zero-shot capability: models learn to recognize unanswerable questions without ever seeing a human-labeled negative during training.
Conclusion and Future Work
SCENE provides a scalable, automated alternative to expensive human annotation for negative examples. While it introduces additional computational overhead during training (approximately 4.5x slower), the gains in robustness and generalization are significant. Future research may focus on optimizing the generator to specifically target the weaknesses of the QA model, further refining the quality of the synthesized counterfactuals.
Summary updated