SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples

Deqing Fu, Ameya Godbole, Robin Jia

Introduction

Natural Language Understanding (NLU) models, particularly those designed for Question Answering (QA), often exhibit an "eager to please" bias. When presented with a passage and a question, these models frequently attempt to extract an answer even when the question is unanswerable based on the provided text. To mitigate this, researchers typically rely on manually curated "negative" examples—unanswerable questions or non-entailed hypotheses. However, manual collection is prohibitively expensive and often introduces human biases that models can exploit as shortcuts.

Furthermore, simple negative examples (e.g., pairing a history question with a biology passage) fail to challenge the model effectively. Robustness requires "hard" negatives: counterfactuals that are linguistically similar to answerable questions but contain subtle semantic contradictions or unmentioned entities. This paper introduces SCENE (Self-labeled Counterfactuals for Extrapolating to Negative Examples), an automated pipeline that synthesizes these challenging examples zero-shot during the training process.

The SCENE Pipeline

SCENE employs a "perturb-filter-label" loop to extrapolate from positive-only data to a new class of negative labels. The pipeline consists of three primary stages:

  1. Perturb: Starting with a positive example, SCENE uses a BART model to mask and fill parts of the text, creating a counterfactual. For instance, a question about "resistant dormant structures" might be perturbed to ask about "contagious strains."
  2. Filter: To ensure quality, the system applies two checks. First, a Paraphrase Detector determines if the new question is semantically identical to the original. Second, Model Consistency is evaluated; if the current QA model changes its answer for a paraphrase, the example is discarded as ambiguous.
  3. Self-Label: If a perturbation is confirmed as a non-paraphrase and the model predicts it is unanswerable, SCENE adopts that label. This "hard negative" is then used to further refine the model.
The SCENE automated pipeline for generating hard negative examples through a perturb-filter-label loop.

This process occurs dynamically during training. As the model improves, its self-labeling becomes more accurate, creating a virtuous cycle of performance gains without human intervention.

Experimental Results

The researchers evaluated SCENE across several NLU tasks, including Extractive QA (SQuAD), Boolean QA (BoolQ), and Recognizing Textual Entailment (RTE). A key finding was the method's ability to perform extrapolation—creating a negative class the model never encountered in its initial training set.

On SQuAD 2.0, SCENE closed 69.6% of the performance gap between a model trained only on positives and an "oracle" model trained on human-annotated data. In Boolean QA, it closed 89.3% of the gap.

Perhaps most significantly, SCENE demonstrated superior generalization on out-of-domain benchmarks. When tested on ACE-whQA, a model trained with SCENE on SQuAD 1.1 (positives only) achieved an F1 score of 73.1, significantly outperforming a model trained on the official human-labeled SQuAD 2.0 dataset, which scored 62.3.

Performance comparison on the ACE-whQA out-of-domain benchmark, showing SCENE's superior generalization compared to human-labeled data.

Key Insights and Limitations

The success of SCENE highlights that the subtlety of counterfactuals is critical. Simple data augmentation, such as shuffling questions and passages, provides minimal benefit. The hard counterfactuals generated by BART force the model to move beyond keyword matching and develop a deeper semantic understanding. Furthermore, the fact that synthetic counterfactuals can outperform human-written ones suggests they may be less susceptible to the annotation artifacts common in manual datasets.

However, the method is not without drawbacks. The inclusion of a BART generator and a paraphrase detector in the training loop increases computational costs by 3x to 4.5x. Additionally, the model requires a "warm-start" on positive data to ensure initial self-labeling is grounded. Future work may explore extending SCENE to multi-class negative labels and integrating small amounts of human adversarial data to further guide the counterfactual generation process.

Summary updated

Abstract

Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging negative examples can help models detect them, it is both costly and domain-specific. In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models’ ability to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels, SCENE can synthesize negative examples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with a mask infilling model, then determines whether the resulting example is negative based on a self-training heuristic. With access to only answerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, a dataset where half of the evaluation examples are unanswerable, compared to a model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.
BibTeX
@inproceedings{fu2023scene,
  author = {Deqing Fu and Ameya Godbole and Robin Jia},
  title = {{SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.485},
}
Figure