Improving Classifier Robustness through Active Generative Counterfactual Data Augmentation

Ananth Balashankar, Xuezhi Wang, Yao Qin, Ben Packer, Nithum Thain, Ed H. Chi, Jilin Chen, Alex Beutel

Abstract

Counterfactual Data Augmentation (CDA) is a commonly used technique for improving robustness in natural language classifiers. However, one fundamental challenge is how to discover meaningful counterfactuals and efficiently label them, with minimal human labeling cost. Most existing methods either completely rely on human-annotated labels, an expensive process which limits the scale of counterfactual data, or implicitly assume label invariance, which may mislead the model with incorrect labels. In this paper, we present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals by actively sampling from regions of uncertainty, and then automatically label them with a learned auxiliary classifier. Our key insight is that we can more correctly label the generated counterfactuals by training a pairwise classifier that interpolates the relationship between the original example and the counterfactual. We demonstrate that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels, that provides an 18-20% improvement in robustness and a 14-21% reduction in errors on 6 out-of-domain datasets, comparable to that of a fully human-annotated counterfactual dataset for both sentiment classification and question paraphrase tasks.

Introduction

Natural Language Processing (NLP) classifiers often exhibit brittleness when faced with minor linguistic variations. For instance, a sentiment analysis model might correctly classify "This movie is great" as positive but fail on "This movie is supposed to be great." To mitigate this, Counterfactual Data Augmentation (CDA) introduces edited versions of training data to improve model generalization. However, CDA is hindered by a "labeling dilemma": human annotation is prohibitively expensive at scale, while automated label-invariance assumptions (assuming the label remains unchanged) are frequently inaccurate. This research addresses the challenge of automatically generating and accurately labeling diverse counterfactuals with minimal human intervention.

Methodology: Active Pairwise Labeling

The proposed framework employs a two-stage pipeline to generate high-quality augmented data.

Active Counterfactual Generation

The system utilizes Polyjuice, a general-purpose counterfactual generator, to produce a variety of edits such as negations, quantifiers, and shuffles. To maximize the utility of these edits, the framework incorporates Active Learning via the Bayesian Active Learning by Disagreement (BALD) algorithm. Instead of random perturbations, the generator specifically targets regions where the current classifier $f$ is most uncertain, ensuring the generated counterfactuals address the model's specific "blind spots."

Pairwise-Counterfactual (PC) Labeling

The core innovation is the Pairwise Classifier ($h$). Rather than labeling an edited sentence $x'$ in isolation, the PC model takes the original sentence $x$, the counterfactual $x'$, and the original label $y$ as inputs to predict the new label $y'$.

The proposed pipeline for active generative counterfactual data augmentation, highlighting the pairwise classifier.

An advanced variant, Classifier-Aware PC (CAPC), further integrates the predictions of the original classifier $f$ on both sentences. This approach leverages the intuition that learning the relationship or "delta" between two similar sentences is a more tractable task than labeling a new sentence from scratch.

Experimental Results

The framework was evaluated on the SST-2 (sentiment analysis) and QQP (question paraphrase) datasets. To assess Out-of-Distribution (OOD) robustness, models trained on SST-2 were tested across six external datasets, including IMDB, Amazon, and Yelp reviews.

Robustness and Accuracy

The results indicate that the CAPC method significantly outperforms standard CDA baselines. Notably, using only 10% human-annotated labels to train the pairwise classifier allowed the system to generate labels for the remaining 90% of data that yielded performance comparable to a fully human-annotated dataset.

Comparison of counterfactual and original accuracy across different augmentation strategies for SST-2 and QQP.

The method provided an 18-20% improvement in robustness on counterfactual test sets and a 14-21% reduction in error rates on OOD datasets. For example, the error rate on Yelp reviews decreased from 15.7% to 10.4%. Crucially, these gains in robustness were achieved without degrading performance on the original "clean" test data.

Conclusion and Future Work

This research demonstrates that active sampling combined with pairwise labeling can drastically reduce the human cost of creating robust NLP models. By focusing on the relationship between original and edited text, the CAPC model provides high-fidelity labels for machine-generated counterfactuals.

Future work may address the limitations of this study, such as its current focus on English and the potential for "label drift" if the auxiliary classifier's errors accumulate. Additionally, exploring more nuanced cases where human annotators disagree could further refine the pairwise labeling process.

Summary updated

Abstract

Counterfactual Data Augmentation (CDA) is a commonly used technique for improving robustness in natural language classifiers. However, one fundamental challenge is how to discover meaningful counterfactuals and efficiently label them, with minimal human labeling cost. Most existing methods either completely rely on human-annotated labels, an expensive process which limits the scale of counterfactual data, or implicitly assume label invariance, which may mislead the model with incorrect labels. In this paper, we present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals by actively sampling from regions of uncertainty, and then automatically label them with a learned auxiliary classifier. Our key insight is that we can more correctly label the generated counterfactuals by training a pairwise classifier that interpolates the relationship between the original example and the counterfactual. We demonstrate that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels, that provides an 18-20% improvement in robustness and a 14-21% reduction in errors on 6 out-of-domain datasets, comparable to that of a fully human-annotated counterfactual dataset for both sentiment classification and question paraphrase tasks.
BibTeX
@inproceedings{balashankar2023improving,
  author = {Ananth Balashankar and Xuezhi Wang and Yao Qin and Ben Packer and Nithum Thain and Ed H. Chi and Jilin Chen and Alex Beutel},
  title = {{Improving Classifier Robustness through Active Generative Counterfactual Data Augmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=PArzHVDlUG},
}
Figure