Improving Classifier Robustness through Active Generation of Pairwise Counterfactuals

Ananth Balashankar, Xuezhi Wang, Yao Qin, Ben Packer, Nithum Thain, Jilin Chen, Ed H. Chi, Alex Beutel

Abstract

Counterfactual Data Augmentation (CDA) is a commonly used technique for improving robustness in natural language classifiers. However, one fundamental challenge is how to discover meaningful counterfactuals and efficiently label them, with minimal human labeling cost. Most existing methods either completely rely on human-annotated labels, an expensive process which limits the scale of counterfactual data, or implicitly assume label invariance, which may mislead the model with incorrect labels. In this paper, we present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals by actively sampling from regions of uncertainty, and then automatically label them with a learned pairwise classifier. Our key insight is that we can more correctly label the generated counterfactuals by training a pairwise classifier that interpolates the relationship between the original example and the counterfactual. We demonstrate that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels, that provides an 18-20% improvement in robustness and a 14-21% reduction in errors on 6 out-of-domain datasets, comparable to that of a fully human-annotated counterfactual dataset for both sentiment classification and question paraphrase tasks.

Introduction

Machine learning classifiers often rely on "spurious correlations"—shortcuts in the data that do not represent the actual logic of a task. For example, a sentiment model might associate the word "Spielberg" with "Positive" simply because most Spielberg movies in its training set were hits. When faced with a rare negative review of a Spielberg film, the model fails.

Counterfactual Data Augmentation (CDA) addresses this by creating "what-if" scenarios: "This movie is great" becomes "This movie is not great." However, CDA faces a significant "labeling bottleneck." Researchers must either pay humans to label thousands of new sentences (expensive) or assume the label stays the same (often false). In NLP, a tiny change—like adding the word "supposedly"—can completely flip a sentence's meaning. This paper introduces a framework to discover meaningful counterfactuals and label them accurately with minimal human intervention.

The Proposed Framework

The authors propose a two-stage pipeline to scale up CDA efficiently by combining generative modeling with a novel labeling strategy.

Overview of the proposed framework for active counterfactual generation and pairwise labeling.
  1. Active Counterfactual Generation: Instead of generating random variations, the framework uses a generative model (Polyjuice) to create diverse counterfactuals. It employs Active Learning (specifically the BALD algorithm) to sample examples from "regions of uncertainty"—places where the current model is most confused. This ensures the generated data is highly informative for the model's training.
  2. Pairwise Counterfactual (PC) Classifier: Instead of training a model to label a single sentence from scratch, the framework uses an auxiliary classifier ($h$) that evaluates the pair: the original sentence, the original label, and the new counterfactual sentence. An advanced version, CAPC (Classifier-Aware PC), also incorporates the base classifier's predictions. The core insight is that it is significantly easier for a model to learn how a label changes (the "delta") than to learn the absolute label of a brand-new sentence in isolation.

Experimental Setup

The framework was evaluated on two primary tasks: Sentiment Classification using the Stanford Sentiment Treebank (SST-2) and Question Paraphrasing using the Quora Question Pair (QQP) dataset. To test real-world robustness, the models were evaluated on six out-of-distribution (OOD) datasets, including reviews from IMDB, Amazon, Yelp, and Twitter (SemEval), which were not seen during training.

Key Results and Insights

The experimental results demonstrate the effectiveness of the pairwise approach:

  • Robustness and Generalization: The method provided an 18-20% improvement in robustness on counterfactual test sets and reduced errors by 14-21% across six out-of-domain datasets.
  • Cost Efficiency: A model trained on 10% human labels and 90% machine labels performed nearly as well as a model trained on a 100% human-annotated counterfactual dataset. This suggests that labeling is easier in context; by providing "before" and "after" versions, the pairwise classifier identifies patterns of label-flipping that standard classifiers miss.
  • Strategic Sampling: By actively seeking out sentences the model is unsure about, the generator creates a "harder" and more useful training set than random perturbation would.

Limitations and Future Work

While promising, the study has limitations. It was restricted to English-language perturbations, and the system's success depends on the quality of the underlying text generator. Low-quality or ungrammatical generations could degrade performance. Additionally, generative models carry risks of perpetuating social biases. Future work will explore more efficient crowdsourcing strategies and the application of this pairwise approach to complex, multi-class classification tasks.

Summary updated

Abstract

Counterfactual Data Augmentation (CDA) is a commonly used technique for improving robustness in natural language classifiers. However, one fundamental challenge is how to discover meaningful counterfactuals and efficiently label them, with minimal human labeling cost. Most existing methods either completely rely on human-annotated labels, an expensive process which limits the scale of counterfactual data, or implicitly assume label invariance, which may mislead the model with incorrect labels. In this paper, we present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals by actively sampling from regions of uncertainty, and then automatically label them with a learned pairwise classifier. Our key insight is that we can more correctly label the generated counterfactuals by training a pairwise classifier that interpolates the relationship between the original example and the counterfactual. We demonstrate that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels, that provides an 18-20% improvement in robustness and a 14-21% reduction in errors on 6 out-of-domain datasets, comparable to that of a fully human-annotated counterfactual dataset for both sentiment classification and question paraphrase tasks.
BibTeX
@inproceedings{balashankar2023improving,
  author = {Ananth Balashankar and Xuezhi Wang and Yao Qin and Ben Packer and Nithum Thain and Jilin Chen and Ed H. Chi and Alex Beutel},
  title = {{Improving Classifier Robustness through Active Generation of Pairwise Counterfactuals}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=PArzHVDlUG},
}
Figure