Counterfactual Editing for Search Result Explanation

Zhichao Xu, Hemank Lamba, Qingyao Ai, Joel R. Tetreault, Alex Jaimes

Abstract

Recently substantial improvements in neural retrieval methods also bring to light the inherent blackbox nature of these methods, especially when viewed from an explainability perspective. Most of existing works on Search Result Explanation (SeRE) are designed to provide factual explanation, i.e. to find/generate supporting evidence about documents' relevance to search queries. However, research in cognitive sciences have shown that human explanations are contrastive i.e. people explain an observed event using some counterfactual events; such explanations reduce cognitive load, and provide actionable insights. Though already proven effective in machine learning and NLP communities, the formulation and impact of counterfactual explanations have not been well studied for search systems. In this work, we aim to investigate the effectiveness of this perspective via proposing and evaluating counterfactual explanations for the task of SeRE. Specifically, we first conduct a user study where we investigate if counterfactual explanations indeed improve search sessions' effectiveness. Taking this as a motivation, we discuss the desiderata that an ideal counterfactual explanation method for SeRE should adhere to. Next, we propose a method $\text{CFE}^2$ (CounterFactual Explanation with Editing) to provide pairwise explanations to search engine result page. Finally, we showcase that the proposed method when evaluated on four publicly available datasets outperforms baselines on both metrics and human evaluation.

Introduction

Modern search engines rely heavily on complex neural retrieval models that, while effective, operate as "black boxes." Users are often left wondering why a specific document was ranked above another. Traditional Search Result Explanation (SeRE) methods attempt to solve this by providing factual evidence, such as highlighting relevant snippets. However, cognitive science suggests that humans prefer contrastive explanations—understanding why outcome A occurred instead of outcome B.

This paper introduces CFE2 (CounterFactual Editing for Search Result Explanation), a framework that provides actionable, contrastive explanations by showing users how a minimal edit to their query would have changed the search results. By focusing on query editing rather than document modification, CFE2 provides insights into the only part of the search process the user directly controls.

The CFE2 Framework

CFE2 is designed to be model-agnostic, meaning it can explain the outputs of any retrieval system without requiring access to its internal parameters. The framework consists of three primary components working in concert:

  1. Search Model ($S$): The underlying retrieval system being explained.
  2. Masker Model ($M$): This component identifies the most influential tokens in the original query. It assigns importance scores to determine which parts of the query are responsible for the current ranking.
  3. Editor Model ($E$): A language model (such as DistilBERT) that suggests alternative tokens for the masked portions of the query.
The Proposed CFE2 framework overview, illustrating the interaction between the Search Model, Masker Model, and Editor Model to generate counterfactual queries.

The Editing Algorithm

The system operates through an iterative process. It begins by masking the single most important token in the query. The Editor Model then generates a beam of candidate replacements. CFE2 checks if any of these candidates "flip" the ranking—making the target document rank higher than the original top result. If no single-token edit succeeds, the algorithm proceeds to mask two tokens, and so on. To ensure the final explanation is natural and useful, the system selects the successful candidate with the highest fluency (lowest perplexity).

Comparative Advantages

CFE2 is built upon several key desiderata for effective search explanations: model agnosticism, counterfactual nature, and minimal modification. Unlike previous methods that might generate entirely new queries or focus only on factual snippets, CFE2 ensures the explanation remains close to the user's original intent.

Comparison of CFE2 with existing explanation methods across key properties such as model agnosticism and minimal modification.

As shown in the comparison table, CFE2 is the only method that satisfies all identified properties for ideal counterfactual explanations in search, maintaining a medium level of computational complexity suitable for near real-time applications.

Experimental Evaluation

The researchers evaluated CFE2 across five diverse datasets: MS MARCO, Natural Questions (NQ), FiQA (finance), SciFact (science), and FEVER (fact verification).

Key Results

  • FlipRate: CFE2 achieved a near 100% success rate in finding valid counterfactual queries across all tested datasets.
  • Human Evaluation: In side-by-side comparisons, human judges rated CFE2 significantly higher than baselines like GenEx in terms of fluency, clarity, and semantic/syntactic closeness to the original query.
  • Domain Adaptation: A variant called CFE2+, which utilizes Task-Adaptive Pre-Training (TAPT), showed even better performance in specialized domains by suggesting more contextually appropriate terminology.

Conclusion and Future Work

CFE2 represents a significant step toward transparent and interactive search systems. By providing contrastive, minimal, and fluent query edits, it empowers users to understand and refine their information-seeking behavior. While the current work focuses on pairwise explanations (comparing two documents), future research will explore listwise explanations to provide a holistic view of the entire search engine result page. Testing these counterfactuals in live, real-world search environments remains a primary goal for future deployment.

Summary updated

Abstract

Recently substantial improvements in neural retrieval methods also bring to light the inherent blackbox nature of these methods, especially when viewed from an explainability perspective. Most of existing works on Search Result Explanation (SeRE) are designed to provide factual explanation, i.e. to find/generate supporting evidence about documents' relevance to search queries. However, research in cognitive sciences have shown that human explanations are contrastive i.e. people explain an observed event using some counterfactual events; such explanations reduce cognitive load, and provide actionable insights. Though already proven effective in machine learning and NLP communities, the formulation and impact of counterfactual explanations have not been well studied for search systems. In this work, we aim to investigate the effectiveness of this perspective via proposing and evaluating counterfactual explanations for the task of SeRE. Specifically, we first conduct a user study where we investigate if counterfactual explanations indeed improve search sessions' effectiveness. Taking this as a motivation, we discuss the desiderata that an ideal counterfactual explanation method for SeRE should adhere to. Next, we propose a method $\text{CFE}^2$ (\textbf{C}ounter\textbf{F}actual \textbf{E}xplanation with \textbf{E}diting) to provide pairwise explanations to search engine result page. Finally, we showcase that the proposed method when evaluated on four publicly available datasets outperforms baselines on both metrics and human evaluation.
BibTeX
@inproceedings{xu2024counterfactual,
  author = {Zhichao Xu and Hemank Lamba and Qingyao Ai and Joel R. Tetreault and Alex Jaimes},
  title = {{Counterfactual Editing for Search Result Explanation}},
  booktitle = {ICTIR},
  year = {2024},
  doi = {10.1145/3664190.3672508},
}
Figure