Mitigating Frequency Bias in Next-Basket Recommendation via Deconfounders

Xiaohan Li, Zheng Liu, Luyi Ma, Kaushiki Nag, Stephen Guo, Philip S. Yu, Kannan Achan

Abstract

Recent progress in Next-basket Recommendation (NBR) has relied heavily on Personalized Item Frequency (PIF). However, this creates a "frequency bias," where systems over-recommend items a user has purchased frequently in the past, narrowing their browsing scope and reducing system utility. We propose FENDER (Frequency-aware Deconfounder for Next-basket Recommendation), a causal inference-based model that treats user and item representations as unobserved confounders. By decomposing PIF into substitute confounders, FENDER provides more diverse and accurate recommendations. Experimental results across three large datasets demonstrate that FENDER outperforms ten baseline models in both accuracy and diversity.

Introduction

In the retail sector, Next-Basket Recommendation (NBR) systems aim to predict the set of items a user will purchase in their next transaction. While effective, many current models suffer from a "filter bubble" effect. By over-relying on Personalized Item Frequency (PIF)—the historical frequency of a user's interactions with an item—these systems become biased toward frequent purchases. This "frequency bias" prevents the discovery of new interests and reduces the overall utility of the recommendation system.

The Causal Inference Problem

The authors frame frequency bias as a causal inference challenge. In a standard recommendation setting, unobserved factors such as a user's underlying personality or an item's intrinsic appeal act as confounders. These factors influence both the historical purchase frequency ($P_t$) and the future basket composition ($R_t$).

Causal diagram showing unobserved confounders (U and I) influencing both past purchase frequency (Pt) and future recommendations (Rt).

Without accounting for these hidden variables, models erroneously assume that past frequency is the direct cause of future behavior, leading to a self-reinforcing loop of repetitive recommendations. This feedback loop trap means models aren't just reflecting reality; they are amplifying existing biases.

Methodology: The FENDER Framework

To address this, the researchers developed FENDER (Frequency-aware Deconfounder for Next-basket Recommendation). The model employs a two-stage deconfounding process:

  1. PIF Decomposition: A Neural Tensor Layer (NTL) factorizes historical PIF data into latent user and item embeddings, capturing long-term preference patterns.
  2. Next-Basket Prediction: The model learns separate embeddings to represent unobserved confounders (current interests).
  3. Deconfounding Inference: By combining these signals with a tunable weight parameter ($\omega$), FENDER can balance conservative (habit-based) and diverse (interest-based) recommendations.

Experimental Evaluation

FENDER was tested on three major datasets: Instacart (3M+ orders), Dunnhumby (in-store transactions), and a proprietary Walmart Grocery dataset. The study focused on users with a history of at least five baskets to properly evaluate long-term bias. To measure success, the authors introduced the negative Top-Frequency Ratio (nTFR), which specifically tracks how many non-frequent items successfully reach the top of the recommendation list.

Results and Performance

FENDER consistently outperformed ten baseline models, including MIT-GNN and Sets2Sets. It achieved an average improvement of 10.20% in NDCG@20, demonstrating that mitigating bias does not have to come at the cost of accuracy.

Average ranking performance of FENDER compared to several baseline models, illustrating its superior recommendation quality.

Furthermore, FENDER showed significant improvements in diversity (nTFR) and demonstrated robustness against "noise" items in user history, correctly identifying them as irrelevant rather than frequent interests. This allows the system to break the "repeat purchase" cycle while maintaining high precision.

Conclusion

FENDER provides a robust framework for breaking the repetitive cycle of frequency-biased recommendations. By leveraging causal deconfounder theory, it balances historical habits with potential new interests. While the current model relies on pre-calculated PIF features, future work will explore integrating more complex sequential patterns, such as seasonality and "running out" cycles, into the deconfounding process.

Summary updated

Abstract

Recent studies on Next-basket Recommendation (NBR) have achieved much progress by leveraging Personalized Item Frequency (PIF) as one of the main features, which measures the frequency of the user's interactions with the item. However, taking the PIF as an explicit feature incurs bias towards frequent items. Items that a user purchases frequently are assigned higher weights in the PIF-based recommender system and appear more frequently in the personalized recommendation list. As a result, the system will lose the fairness and balance between items that the user frequently purchases and items that the user never purchases. We refer to this systematic bias on personalized recommendation lists as frequency bias, which narrows users' browsing scope and reduces the system utility. We adopt causal inference theory to address this issue. Considering the influence of historical purchases on users' future interests, the user and item representations can be viewed as unobserved confounders in the causal diagram. In this paper, we propose a deconfounder model named FENDER (Frequency-aware Deconfounder for Next-basket Recommendation) to mitigate the frequency bias. With the deconfounder theory and the causal diagram we propose, FENDER decomposes PIF with a neural tensor layer to obtain substitute confounders for users and items. Then, FENDER performs unbiased recommendations considering the effect of these substitute confounders. Experimental results demonstrate that FENDER has derived diverse and fair results compared to ten baseline models on three datasets while achieving competitive performance. Further experiments illustrate how FENDER balances users' historical purchases and potential interests.
BibTeX
@inproceedings{li2022mitigating,
  author = {Xiaohan Li and Zheng Liu and Luyi Ma and Kaushiki Nag and Stephen Guo and Philip S. Yu and Kannan Achan},
  title = {{Mitigating Frequency Bias in Next-Basket Recommendation via Deconfounders}},
  booktitle = {IEEE Big Data},
  year = {2022},
  doi = {10.1109/bigdata55660.2022.10021135},
}
Figure