SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples

Deqing Fu, Ameya Godbole, Robin Jia

TL;DR

SCENE (Self-labeled Counterfactuals for Extrapolating to Negative Examples) is an automated pipeline that synthesizes challenging "negative" training data—such as unanswerable questions or non-entailing sentences—from purely "positive" examples. By combining counterfactual perturbations with a self-training heuristic, SCENE allows models to learn to detect subtle negatives zero-shot, closing nearly 70% of the performance gap on SQuAD 2.0 without requiring a single human-annotated negative example.

What Problem Does This Paper Address?

In Natural Language Understanding (NLU), models are often great at finding answers (positives) but terrible at knowing when an answer doesn't exist (negatives). For instance, in extractive Question Answering (QA), a model might confidently provide a wrong answer to a question that is actually unanswerable based on the provided text.

While datasets like SQuAD 2.0 include human-written unanswerable questions to train against this, collecting such "hard negatives" is expensive, time-consuming, and often introduces human biases. Simple automated methods, like pairing a question with a random passage ("shuffling"), produce negatives that are too easy for modern models to solve, failing to teach them the fine-grained distinctions needed for real-world robustness. The authors seek a way to automatically generate subtle negative examples that force the model to understand deeper semantic nuances.

Methods

The authors propose a two-stage approach. First, they "warm-start" the model using simple negatives generated via Shuffle-Based and Retrieval-Based methods (pairing questions with passages that have high word overlap but no answer).

Second, they introduce the SCENE pipeline to generate harder, counterfactual examples:

  1. Perturb: The system takes a positive question and masks random tokens. A mask-denoising model (BART) fills these masks to create a "counterfactual" question that is syntactically similar but semantically different.
  2. Filter: To ensure quality, SCENE uses a paraphrase detector and the current QA model's own predictions as a "rejection sampler." If the perturbation is too ambiguous (e.g., the model's prediction changes but the paraphrase detector says the meaning is the same), the example is discarded.
  3. Self-Label: If the example passes the filter, the model trusts its own prediction to label the new pair. Usually, a meaning-altering perturbation results in a "No Answer" label.
The SCENE pipeline: (1) perturbation, (2) paraphrase detection, (3) model prediction, and (4) self-labeling/filtering.

Crucially, this happens dynamically during training; the model acts as both the teacher (labeling new data) and the student.

Datasets

The researchers evaluated SCENE across three distinct NLU tasks:

  • Extractive QA: Training on SQuAD 1.1 (positives only) and testing on SQuAD 2.0 (which adds unanswerable questions) and ACE-whQA (an out-of-domain benchmark).
  • Boolean QA: Training on BoolQ (Yes/No) and testing on BoolQ-3L (Yes/No/IDK).
  • Recognizing Textual Entailment (RTE): Using the entailment-only subset of the RTE dataset for training.

Key Insights and Ideas

  • Extrapolation vs. Augmentation: Traditional data augmentation creates more examples for existing labels. SCENE performs extrapolation, enabling a model to learn a completely new class (the negative class) that was never present in the original gold-standard training set.
  • The Power of Subtlety: Shuffling passages (simple negatives) barely improves performance on SQuAD 2.0. The "aha moment" is that models need counterfactuals—questions that differ by only a word or two (e.g., changing "resistant dormant structure" to "contagious strains")—to truly learn the boundaries of answerability.
  • Superior Generalization: Surprisingly, a model trained on SQuAD 1.1 + SCENE outperformed a model trained on the "gold" SQuAD 2.0 dataset when tested on out-of-domain data (ACE-whQA). This suggests that synthetic counterfactuals may be less prone to the specific biases found in human-written negatives.

Key Results

SCENE demonstrated significant gains across all tasks, effectively "closing the gap" between models trained only on positives and those trained on human-annotated negatives:

  • SQuAD 2.0: SCENE closed 69.6% of the performance gap. While a baseline SQuAD 1.1 model achieved only 45.7 F1, the SCENE-enhanced version reached 71.8 F1.
  • Boolean QA: The method closed 89.3% of the gap on BoolQ-3L, reaching 78.1% accuracy.
  • RTE: SCENE closed 56.1% of the gap, improving accuracy from 52.7% to 67.9%.
  • Out-of-Domain: On ACE-whQA, SCENE achieved 73.1 F1, notably higher than the 62.3 F1 achieved by the "oracle" model trained on human-annotated SQuAD 2.0 data.
Table 2: Evaluation on SQuAD 2.0. Training on positive-only SQuAD 1.1, SCENE with Retrieval can close 69.6% of the gap compared with an oracle model trained on SQuAD 2.0. We report the mean and standard deviation (in parentheses) over 3 runs. Best scores among all methods that train on SQuAD 1.1 are highlighted in bold.

Limitations and Future Work

The authors acknowledge that SCENE comes with a higher computational cost; generating and filtering examples during every training batch is slower than standard training. Additionally, the BART-based generator is not specifically tuned to produce certain types of hard negatives, like antonym substitutions or negations, which are common in human-written datasets.

Future work includes exploring ways to back-propagate signals from the filtering process directly into the generator so it can learn to produce even more useful perturbations. The authors also suggest combining this automatic pipeline with small amounts of human-adversarial data to further refine the quality of the synthetic examples.

Summary updated

Abstract

Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging negative examples can help models detect them, it is both costly and domain-specific. In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models' ability to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels, SCENE can synthesize negative examples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with a mask infilling model, then determines whether the resulting example is negative based on a self-training heuristic. With access to only answerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, a dataset where half of the evaluation examples are unanswerable, compared to a model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.
BibTeX
@inproceedings{fu2023scene,
  author = {Deqing Fu and Ameya Godbole and Robin Jia},
  title = {{SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.485},
}
Figure