Introduction
Natural Language Understanding (NLU) models, particularly those designed for Question Answering (QA), often exhibit an "eager to please" bias. When presented with a passage and a question, these models frequently attempt to extract an answer even when the question is unanswerable based on the provided text. To mitigate this, researchers typically rely on manually curated "negative" examples—unanswerable questions or non-entailed hypotheses. However, manual collection is prohibitively expensive and often introduces human biases that models can exploit as shortcuts.
Furthermore, simple negative examples (e.g., pairing a history question with a biology passage) fail to challenge the model effectively. Robustness requires "hard" negatives: counterfactuals that are linguistically similar to answerable questions but contain subtle semantic contradictions or unmentioned entities. This paper introduces SCENE (Self-labeled Counterfactuals for Extrapolating to Negative Examples), an automated pipeline that synthesizes these challenging examples zero-shot during the training process.
The SCENE Pipeline
SCENE employs a "perturb-filter-label" loop to extrapolate from positive-only data to a new class of negative labels. The pipeline consists of three primary stages:
- Perturb: Starting with a positive example, SCENE uses a BART model to mask and fill parts of the text, creating a counterfactual. For instance, a question about "resistant dormant structures" might be perturbed to ask about "contagious strains."
- Filter: To ensure quality, the system applies two checks. First, a Paraphrase Detector determines if the new question is semantically identical to the original. Second, Model Consistency is evaluated; if the current QA model changes its answer for a paraphrase, the example is discarded as ambiguous.
- Self-Label: If a perturbation is confirmed as a non-paraphrase and the model predicts it is unanswerable, SCENE adopts that label. This "hard negative" is then used to further refine the model.
This process occurs dynamically during training. As the model improves, its self-labeling becomes more accurate, creating a virtuous cycle of performance gains without human intervention.
Experimental Results
The researchers evaluated SCENE across several NLU tasks, including Extractive QA (SQuAD), Boolean QA (BoolQ), and Recognizing Textual Entailment (RTE). A key finding was the method's ability to perform extrapolation—creating a negative class the model never encountered in its initial training set.
On SQuAD 2.0, SCENE closed 69.6% of the performance gap between a model trained only on positives and an "oracle" model trained on human-annotated data. In Boolean QA, it closed 89.3% of the gap.
Perhaps most significantly, SCENE demonstrated superior generalization on out-of-domain benchmarks. When tested on ACE-whQA, a model trained with SCENE on SQuAD 1.1 (positives only) achieved an F1 score of 73.1, significantly outperforming a model trained on the official human-labeled SQuAD 2.0 dataset, which scored 62.3.
Key Insights and Limitations
The success of SCENE highlights that the subtlety of counterfactuals is critical. Simple data augmentation, such as shuffling questions and passages, provides minimal benefit. The hard counterfactuals generated by BART force the model to move beyond keyword matching and develop a deeper semantic understanding. Furthermore, the fact that synthetic counterfactuals can outperform human-written ones suggests they may be less susceptible to the annotation artifacts common in manual datasets.
However, the method is not without drawbacks. The inclusion of a BART generator and a paraphrase detector in the training loop increases computational costs by 3x to 4.5x. Additionally, the model requires a "warm-start" on positive data to ensure initial self-labeling is grounded. Future work may explore extending SCENE to multi-class negative labels and integrating small amounts of human adversarial data to further guide the counterfactual generation process.
Summary updated