SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples

Deqing Fu, Ameya Godbole, Robin Jia

Abstract

Detecting negatives—such as unanswerable questions or false claims—is a critical yet challenging component of natural language understanding (NLU). While human-annotated negative examples are effective, they are expensive to produce and often contain domain-specific biases. We introduce SCENE (Self-labeled Counterfactuals for Extrapolating to Negative Examples), an automated framework that synthesizes challenging negative training data from purely positive examples. By perturbing positive instances and applying a self-training heuristic, SCENE enables models to extrapolate to negative classes zero-shot. Our method closes 69.6% of the performance gap on SQuAD 2.0 compared to models trained on human-labeled negatives and demonstrates strong generalization across boolean QA and textual entailment tasks.

Introduction: The Challenge of Negative Examples

Robust NLU requires models to distinguish between supported claims (positives) and unsupported ones (negatives). In extractive Question Answering (QA), for instance, a model must determine if a question is answerable based on the provided context.

High-quality negative examples are difficult to acquire. Human-annotated datasets like SQuAD 2.0 are costly and often suffer from "annotation artifacts"—simple patterns that models exploit without true comprehension. Conversely, automated methods like "distant supervision" (pairing random questions with unrelated text) create negatives that are too simplistic. SCENE addresses this by generating "subtle" negatives that force models to learn fine-grained semantic distinctions.

Methodology: The SCENE Pipeline

SCENE generates challenging negatives on-the-fly during training through a three-step process:

  1. Perturb: Starting with a positive example, SCENE masks tokens in the question and uses a BART model to infill them. This creates a "counterfactual" question that remains linguistically similar to the original but may no longer be answerable.
  2. Filter: To ensure quality, two filters are applied:
  3. Paraphrase Detector: A RoBERTa-based model ensures the new question isn't a mere paraphrase of the original, which would likely retain the same answer.
  4. Model Consistency: The current QA model's prediction for the perturbed question is compared to the original. If the perturbation fails to change the predicted answer or causes extreme confusion, the example is discarded.
  5. Self-Label: Examples passing these filters are labeled as "unanswerable" (or "not-entailed") and integrated into the training set. This self-training curriculum progressively challenges the model with increasingly difficult negatives.
The SCENE pipeline: Perturb, Filter, and Self-Label.

Key Insights: The Power of Subtle Perturbations

The effectiveness of SCENE lies in its ability to create "subtle" negatives—examples that look nearly identical to positives but contain a small semantic shift (e.g., swapping an entity or changing a verb) that flips the label.

Examples of original vs. perturbed questions and their impact on answerability.

Surprisingly, models trained with SCENE on SQuAD 1.1 (positives only) often generalize better to out-of-domain data (like ACE-whQA) than models trained on human-annotated SQuAD 2.0. This suggests that synthetic counterfactuals can be more diverse and less prone to the biases found in human-written negatives.

Experimental Results

The researchers evaluated SCENE across several benchmarks:

  • Extractive QA: On SQuAD 2.0, SCENE closed 69.6% of the performance gap between a model trained only on positives and one trained on the full human-annotated dataset.
  • Boolean QA: On BoolQ-3L, the method closed 89.3% of the gap, effectively teaching the model to handle "Yes/No/IDK" scenarios.
  • Textual Entailment: For RTE, SCENE closed 56.1% of the gap using only entailment examples as a starting point.

These results demonstrate a powerful zero-shot capability: models learn to recognize unanswerable questions without ever seeing a human-labeled negative during training.

Conclusion and Future Work

SCENE provides a scalable, automated alternative to expensive human annotation for negative examples. While it introduces additional computational overhead during training (approximately 4.5x slower), the gains in robustness and generalization are significant. Future research may focus on optimizing the generator to specifically target the weaknesses of the QA model, further refining the quality of the synthesized counterfactuals.

Summary updated

Abstract

Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging negative examples can help models detect them, it is both costly and domain-specific. In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models' ability to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels, SCENE can synthesize negative examples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with a mask infilling model, then determines whether the resulting example is negative based on a self-training heuristic. With access to only answerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, a dataset where half of the evaluation examples are unanswerable, compared to a model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.
BibTeX
@inproceedings{fu2023scene,
  author = {Deqing Fu and Ameya Godbole and Robin Jia},
  title = {{SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.485},
}
Figure