Counterfactual Profile Perturbations for Recommendation Explanations: A Reproducibility Study

Meysam Varasteh, Elizabeth McKinnie, Robin BurkearXiv 2026 · arXiv

This reproducibility study dives into Learning to eXplain Recommendations (LXR), a model-agnostic framework for generating counterfactual explanations. By extending the original research to broader recommendation tasks and introducing new baselines, the authors provide a reality check on how practical these "if-then" explanations truly are for real-world recommender systems.

TL;DR

This paper reproduces and extends the LXR framework, showing it successfully generalizes to explaining items beyond the top-ranked recommendation. However, it reveals that simple latent factor similarity is often a surprisingly strong baseline and that "truthful" counterfactual explanations are frequently too long for users to easily digest.

What Problem Does This Paper Address?

Most recommender systems are "black boxes"—complex models like Variational Autoencoders (VAEs) that offer no clear reason why a specific item was suggested. Counterfactual explanations aim to fix this by answering: "What would I have to change in my profile for this item NOT to be recommended?"

The original LXR paper proposed a way to find these "perturbations" (items to remove from a user's profile). However, that study only looked at the very top recommendation and lacked simple, intuitive baselines. This paper fills those gaps, asking if LXR works for the whole top-10 list, how it compares to simpler methods, and whether the resulting explanations are actually concise enough for humans.

Methods

The authors follow the LXR evaluation process (illustrated in Figure 1) but introduce several key modifications:

  • LXR10: Instead of training the explainer only on the top-1 item, they train it on a random item sampled from the top 10 recommendations to see if the model generalizes.
  • New Baselines:
    • LF (Latent Factor): A "glass-box" approach that ranks profile items based on their similarity to the target item in the recommender's own latent embedding space.
  • Pop (Popularity): A non-personalized baseline that simply removes the most popular items from a user's profile first.
  • Metrics: They move away from "percentage of profile" metrics to MPNR (Minimum Perturbation to Not Recommend), which counts the absolute number of items removed, and Coverage, the percentage of users for whom a successful explanation can be found.
Figure 1: LXR evaluation process

Datasets

The study uses three standard collaborative filtering datasets, preprocessed to simulate implicit feedback:

  1. MovieLens 1M (ML1M): ~576k ratings.
  2. Yahoo Music: ~365k ratings.
  3. Pinterest: ~486k ratings.

Key Insights and Ideas

  • The "Glass-Box" Surprise: For Matrix Factorization (MF) models, the LF baseline (latent factor similarity) often outperformed LXR. This suggests that if you have access to the model's internal embeddings, you might not need a complex, separate explainer model at all.
  • The Popularity Trap: Popular items are "sticky." Because recommenders are biased toward them, it takes a massive perturbation to "explain them away." The authors found that LXR provides much shorter, higher-quality explanations for "long-tail" (less popular) items.
  • Non-Uniqueness: Different explainers (LXR vs. LF) often pick completely different items to achieve the same result (low RBO overlap). This means there isn't one "true" explanation; rather, there are multiple ways to nudge a recommendation out of the top list.
  • The "Too Long; Didn't Read" Problem: Even the most "optimal" perturbations (found via brute-force search) are often quite large—frequently 7 to 10 items or more.

Key Results

  • Generalization: LXR10 proved highly effective, often matching or beating the original LXR even on the Top-1 task, proving the method is robust across the recommendation list.
  • Coverage vs. Size: As shown in Figure 2, there is a clear trade-off. To explain recommendations for more users (higher coverage), you generally have to accept much larger perturbation sizes (higher MPNR).
  • Optimality: A brute-force search on a subset of data showed that LXR is close to optimal. For example, on ML1M, LXR found a perturbation of size 10 where the absolute mathematical minimum was 7. While LXR is efficient, the "minimal" explanation is still often too long for a simple UI.
Figure 2: Coverage versus minimum perturbation size for the different recommenders and datasets (Top10 task).

Limitations and Future Work

  • Cognitive Load: The authors argue that an explanation like "If you hadn't liked these 12 movies, we wouldn't have suggested Interstellar" is likely useless to a human. Future work needs to focus on summarizing these perturbations (e.g., by genre or cluster).
  • User Validation: This is a purely algorithmic study. The next step is user studies to see if people actually find counterfactual perturbations more helpful than traditional "Because you liked X" explanations.
  • Hybrid Explanations: Since popular items are so hard to explain counterfactually, the authors suggest a "mixed" approach: use LXR for niche items and global, popularity-based explanations for blockbusters.

Summary updated

Abstract

This reproducibility paper explores the counterfactual explanation technique known as Learning to eXplain Recommendations (LXR). Prior work has shown that LXR can reliably identify perturbations that, if applied to a user's profile, cause the user's top ranked item to no longer be recommended, thereby explaining the item's recommendation as a function of the user's rating choices. In this paper, we extend this work in multiple ways. We expand the scope of the original LXR study to look at explanations for items other than the most highly recommended item, finding that LXR is also effective at this version of the task. We consider alternate baselines for LXR, including item popularity and latent factor similarity, and we see that for some tasks these alternatives work surprisingly well. We also examine the impact of item popularity, finding that LXR offers better explanation quality (shorter perturbations) when less popular items are being explained. Finally, we conduct a brute-force exploration of a subset of the perturbation space in order to estimate a lower bound for optimal perturbation size against which LXR and other such methods can be compared.
BibTeX
@misc{varasteh2026counterfactual,
  author = {Meysam Varasteh and Elizabeth McKinnie and Robin Burke},
  title = {{Counterfactual Profile Perturbations for Recommendation Explanations: A Reproducibility Study}},
  year = {2026},
  eprint = {2610.36208},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36208},
}
Figure