iFlip: Iterative Feedback-driven Counterfactual Example Refinement

Yilong Wang, Qianli Wang, Nils Feldhus

TL;DR

iFlip is an iterative framework that treats counterfactual generation as a multi-turn conversation between a generator (LLM) and an explained model. By using feedback signals like model confidence, feature attribution, and natural language critiques, iFlip achieves a 57.8% higher validity in label flipping compared to state-of-the-art single-pass methods.

What Problem Does This Paper Address?

In Explainable AI (XAI), counterfactual examples (CFEs) are "what-if" scenarios: minimal edits to an input that change a model's prediction (e.g., changing "The acting was superb" to "The acting was subpar" to flip a sentiment label from Positive to Negative). These are vital for debugging models and augmenting training data.

However, most current approaches rely on single-pass generation. Large Language Models (LLMs) are prompted once to create a CFE, but they often fail to actually flip the target model's prediction on the first try. These "invalid" counterfactuals are useless for explanation. The authors argue that we are neglecting the inherent self-correction capabilities of LLMs, which perform much better when given a chance to refine their work based on specific feedback.

Methods

The authors introduce iFlip, a three-stage iterative loop:

  1. Generation: An LLM generator produces an initial counterfactual candidate.
  2. Verification: The "explained model" (the classifier we are probing) evaluates the candidate. If the label flips to the target, the process stops (Early Stopping).
  3. Refinement: If the label didn't flip, the system generates a feedback signal and sends the candidate back to the generator for another attempt.

The paper explores three distinct feedback signals:

  • Confidence Feedback: Providing the generator with the model's probability score for the (incorrect) prediction.
  • Feature Attribution Feedback: Using XAI techniques (like SHAP, LIME, AttnLRP, or Grad×Input) to identify the specific words most responsible for the current prediction, essentially telling the LLM: "Change these words to flip the label."
  • Natural Language (NL) Feedback: Using an LLM to generate a text-based critique of why the current counterfactual failed and how to improve it.

Datasets

The framework was evaluated across three diverse NLP tasks:

  • IMDb: Sentiment analysis of movie reviews.
  • AG News: Four-way topic classification of news articles.
  • SNLI: Natural Language Inference (determining if a premise entails, contradicts, or is neutral toward a hypothesis).

The authors used OLMo2-7B, Qwen3-32B, and LLaMA3.3-70B as generators, providing a look at how model scale affects refinement.

Key Insights and Ideas

  • The Power of "Try Again": Iterative refinement is significantly more effective than single-pass generation. Even a few rounds (K=5) drastically improve the Label Flipping Rate (LFR).
  • Natural Language is the Best Teacher: While all feedback types helped, Natural Language feedback was the most effective at guiding the LLM to valid counterfactuals, though it comes with the highest computational cost.
  • Early Stopping is Non-Negotiable: Without early stopping, the LLM often "over-corrects." It might successfully flip a label in round 2, but if forced to continue to round 5, it may edit the text so much that it becomes nonsensical or flips the label back to the original.
  • NLI is the Final Boss: Generating counterfactuals for Natural Language Inference (SNLI) is much harder than sentiment or topic classification. This is because NLI requires maintaining complex logical relationships between two sentences (premise and hypothesis), where a single word change can have cascading logical effects.

Key Results

  • Validity Boost: iFlip outperformed five baselines (including BAE, Polyjuice, and FIZLE) by an average of 57.8% in validity (Label Flipping Rate).
  • Human Preference: In a user study, iFlip-generated counterfactuals saw an 83.21% improvement in overall satisfaction compared to baselines, scoring higher on completeness and feasibility.
  • Data Augmentation: Using iFlip to generate synthetic training data (Counterfactual Data Augmentation) led to measurable improvements in model performance and robustness on out-of-distribution test sets.
  • Efficiency vs. Quality: Confidence-based feedback emerged as a "sweet spot"—it's much faster than NL feedback but still provides a significant boost over no-feedback baselines.

Limitations and Future Work

  • Computational Cost: Iterative loops require multiple LLM calls per example. Using NL feedback effectively doubles the cost per iteration (one call for feedback, one for generation).
  • English-Centric: The study was limited to English datasets; the authors plan to expand to multilingual settings.
  • Attribution Faithfulness: The authors noted that current feature attribution methods (like SHAP) sometimes struggle to identify the "right" words in complex tasks like NLI, leading to less effective feedback.
  • Scale of User Study: The human evaluation was conducted with a relatively small group (N=3), suggesting a need for larger-scale validation in the future.

Summary updated

Abstract

Counterfactual examples are minimal edits to an input that alter a model's prediction. They are widely employed in explainable AI to probe model behavior and in natural language processing (NLP) to augment training data. However, generating valid counterfactuals with large language models (LLMs) remains challenging, as existing single-pass methods often fail to induce reliable label changes, neglecting LLMs' self-correction capabilities. To explore this untapped potential, we propose iFlip, an iterative refinement approach that leverages three types of feedback, including model confidence, feature attribution, and natural language. Our results show that iFlip achieves an average 57.8% higher validity than the five state-of-the-art baselines, as measured by the label flipping rate. The user study further corroborates that iFlip outperforms baselines in completeness, overall satisfaction, and feasibility. In addition, ablation studies demonstrate that three components are paramount for iFlip to generate valid counterfactuals: leveraging an appropriate number of iterations, pointing to highly attributed words, and early stopping. Finally, counterfactuals generated by iFlip enable effective counterfactual data augmentation, substantially improving model performance and robustness.
BibTeX
@misc{wang2026iflip,
  author = {Yilong Wang and Qianli Wang and Nils Feldhus},
  title = {{iFlip: Iterative Feedback-driven Counterfactual Example Refinement}},
  year = {2026},
  eprint = {2601.01446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.01446},
}
Figure