Mitigating Frequency Bias in Next-Basket Recommendation via Deconfounders

Xiaohan Li, Zheng Liu, Luyi Ma, Kaushiki Nag, Stephen D. Guo, Philip S. Yu, Kannan Achan

This paper, a collaboration between researchers at Walmart Global Tech, the University of Illinois at Chicago, and Indeed, tackles a pervasive issue in grocery and retail AI: the "filter bubble" of repetitive shopping.

TL;DR

Next-Basket Recommendation (NBR) systems often suffer from frequency bias, where they over-recommend items a user has bought many times before, failing to suggest new products. The authors propose FENDER, a causal deconfounder framework that disentangles historical habits from true current interest to provide recommendations that are both more accurate and more diverse.

What Problem Does This Paper Address?

In Next-Basket Recommendation (NBR), the goal is to predict the entire "basket" of items a user will buy in their next trip. Most state-of-the-art models rely heavily on Personalized Item Frequency (PIF)—a metric of how often you’ve bought an item in the past.

While PIF is a strong predictor (if you buy milk every week, you’ll likely buy it next week), over-reliance on it creates a "personalized popularity bias." The system becomes a mirror of your past rather than a helpful assistant. It stops discovering your evolving interests and narrows your browsing scope.

The authors argue this is a causal inference problem. Unobserved factors, like a user's "adventurous" or "prudent" personality, act as confounders: they influence both your past purchase frequency and your future choices. If a model doesn't account for these hidden confounders, it erroneously treats past frequency as the sole cause of future behavior, reinforcing stereotypes and reducing the system's utility.

Methods

The authors introduce FENDER (Frequency-aware Deconfounder for Next-basket Recommendation), which uses "deconfounder theory" to mitigate bias through a two-stage training process:

  1. Stage 1: PIF Decomposition: The model uses a Neural Tensor Layer (NTL) to factorize the PIF data into latent user and item embeddings. This stage essentially "learns" the patterns of historical preference.
  2. Stage 2: Next-Basket Prediction: The model learns a separate set of embeddings to represent the unobserved confounders (current interests).
  3. Inference (The "Deconfounding" Step): FENDER combines the signals from both stages using a weight parameter ($\omega$). By adjusting this weight, the system can tune its recommendation style from "conservative" (favoring high-frequency items) to "diverse" (favoring potential new interests).

This approach allows the model to treat the historical frequency as just one piece of the puzzle, rather than the definitive answer.

Datasets

The researchers evaluated FENDER on three large-scale grocery datasets:

  • Instacart: A public dataset of over 3 million online grocery orders.
  • Dunnhumby: Real-world in-store transaction data.
  • Walmart Grocery: A proprietary dataset from Walmart Inc., representing a portion of their massive monthly active user base.

To focus on long-term bias, they specifically selected users with more than five baskets of history.

Key Insights and Ideas

  • The Feedback Loop Trap: The authors show that standard models (like TIFU-KNN) actually recommend repeat items more often than they appear in the ground-truth data. This proves that models aren't just reflecting reality; they are amplifying bias.
  • Substitute Confounders: Since we can't directly measure a user's "personality" or "mood," FENDER uses the latent factors learned from the PIF decomposition as "substitute confounders." This is a clever application of causal theory to a domain where "ground truth" causes are invisible.
  • A New Metric (nTFR): They introduce the negative Top-Frequency Ratio (nTFR). While standard metrics like Hit Rate measure accuracy, nTFR specifically measures diversity by looking at how many non-frequent items make it into the top recommendations.

Key Results

  • Superior Accuracy: FENDER outperformed ten baseline models (including popular ones like BPR-MF and MIT-GNN) across all three datasets. It achieved an average improvement of 10.20% in NDCG@20 (a measure of ranking quality).
  • Better Diversity: FENDER significantly improved diversity scores (nTFR), especially on the Walmart and Dunnhumby datasets, proving it can break the "repeat purchase" cycle without losing accuracy.
  • Robustness to Noise: In a "stress test," the researchers manually inserted random "noise" items into users' histories. While PIF-based models were fooled into recommending these irrelevant items at the top of the list, FENDER correctly identified them as noise and ranked them much lower.

Limitations and Future Work

The authors acknowledge that while FENDER is effective, it relies on a pre-calculated PIF feature. This simplifies the computation but might overlook more complex, time-sensitive sequential patterns (e.g., seasonal changes or "running out" cycles).

Future work could involve extending this deconfounder approach to more complex sequential models or exploring how to substitute other modules in the recommendation pipeline to handle different types of systemic bias beyond just frequency.

Summary updated

Abstract

Recent studies on Next-basket Recommendation (NBR) have achieved much progress by leveraging Personalized Item Frequency (PIF) as one of the main features, which measures the frequency of the user's interactions with the item. However, taking the PIF as an explicit feature incurs bias towards frequent items. Items that a user purchases frequently are assigned higher weights in the PIF-based recommender system and appear more frequently in the personalized recommendation list. As a result, the system will lose the fairness and balance between items that the user frequently purchases and items that the user never purchases. We refer to this systematic bias on personalized recommendation lists as frequency bias, which narrows users' browsing scope and reduces the system utility. We adopt causal inference theory to address this issue. Considering the influence of historical purchases on users' future interests, the user and item representations can be viewed as unobserved confounders in the causal diagram. In this paper, we propose a deconfounder model named FENDER (Frequency-aware Deconfounder for Next-basket Recommendation) to mitigate the frequency bias. With the deconfounder theory and the causal diagram we propose, FENDER decomposes PIF with a neural tensor layer to obtain substitute confounders for users and items. Then, FENDER performs unbiased recommendations considering the effect of these substitute confounders. Experimental results demonstrate that FENDER has derived diverse and fair results compared to ten baseline models on three datasets while achieving competitive performance. Further experiments illustrate how FENDER balances users' historical purchases and potential interests.
BibTeX
@inproceedings{li2022mitigating,
  author = {Xiaohan Li and Zheng Liu and Luyi Ma and Kaushiki Nag and Stephen D. Guo and Philip S. Yu and Kannan Achan},
  title = {{Mitigating Frequency Bias in Next-Basket Recommendation via Deconfounders}},
  booktitle = {IEEE Big Data},
  year = {2022},
  doi = {10.1109/bigdata55660.2022.10021135},
}

Loading related content...

Figure