This paper, a collaboration between researchers at Walmart Global Tech, the University of Illinois at Chicago, and Indeed, tackles a pervasive issue in grocery and retail AI: the "filter bubble" of repetitive shopping.
TL;DR
Next-Basket Recommendation (NBR) systems often suffer from frequency bias, where they over-recommend items a user has bought many times before, failing to suggest new products. The authors propose FENDER, a causal deconfounder framework that disentangles historical habits from true current interest to provide recommendations that are both more accurate and more diverse.
What Problem Does This Paper Address?
In Next-Basket Recommendation (NBR), the goal is to predict the entire "basket" of items a user will buy in their next trip. Most state-of-the-art models rely heavily on Personalized Item Frequency (PIF)—a metric of how often you’ve bought an item in the past.
While PIF is a strong predictor (if you buy milk every week, you’ll likely buy it next week), over-reliance on it creates a "personalized popularity bias." The system becomes a mirror of your past rather than a helpful assistant. It stops discovering your evolving interests and narrows your browsing scope.
The authors argue this is a causal inference problem. Unobserved factors, like a user's "adventurous" or "prudent" personality, act as confounders: they influence both your past purchase frequency and your future choices. If a model doesn't account for these hidden confounders, it erroneously treats past frequency as the sole cause of future behavior, reinforcing stereotypes and reducing the system's utility.
Methods
The authors introduce FENDER (Frequency-aware Deconfounder for Next-basket Recommendation), which uses "deconfounder theory" to mitigate bias through a two-stage training process:
- Stage 1: PIF Decomposition: The model uses a Neural Tensor Layer (NTL) to factorize the PIF data into latent user and item embeddings. This stage essentially "learns" the patterns of historical preference.
- Stage 2: Next-Basket Prediction: The model learns a separate set of embeddings to represent the unobserved confounders (current interests).
- Inference (The "Deconfounding" Step): FENDER combines the signals from both stages using a weight parameter ($\omega$). By adjusting this weight, the system can tune its recommendation style from "conservative" (favoring high-frequency items) to "diverse" (favoring potential new interests).
This approach allows the model to treat the historical frequency as just one piece of the puzzle, rather than the definitive answer.
Datasets
The researchers evaluated FENDER on three large-scale grocery datasets:
- Instacart: A public dataset of over 3 million online grocery orders.
- Dunnhumby: Real-world in-store transaction data.
- Walmart Grocery: A proprietary dataset from Walmart Inc., representing a portion of their massive monthly active user base.
To focus on long-term bias, they specifically selected users with more than five baskets of history.
Key Insights and Ideas
- The Feedback Loop Trap: The authors show that standard models (like TIFU-KNN) actually recommend repeat items more often than they appear in the ground-truth data. This proves that models aren't just reflecting reality; they are amplifying bias.
- Substitute Confounders: Since we can't directly measure a user's "personality" or "mood," FENDER uses the latent factors learned from the PIF decomposition as "substitute confounders." This is a clever application of causal theory to a domain where "ground truth" causes are invisible.
- A New Metric (nTFR): They introduce the negative Top-Frequency Ratio (nTFR). While standard metrics like Hit Rate measure accuracy, nTFR specifically measures diversity by looking at how many non-frequent items make it into the top recommendations.
Key Results
- Superior Accuracy: FENDER outperformed ten baseline models (including popular ones like BPR-MF and MIT-GNN) across all three datasets. It achieved an average improvement of 10.20% in NDCG@20 (a measure of ranking quality).
- Better Diversity: FENDER significantly improved diversity scores (nTFR), especially on the Walmart and Dunnhumby datasets, proving it can break the "repeat purchase" cycle without losing accuracy.
- Robustness to Noise: In a "stress test," the researchers manually inserted random "noise" items into users' histories. While PIF-based models were fooled into recommending these irrelevant items at the top of the list, FENDER correctly identified them as noise and ranked them much lower.
Limitations and Future Work
The authors acknowledge that while FENDER is effective, it relies on a pre-calculated PIF feature. This simplifies the computation but might overlook more complex, time-sensitive sequential patterns (e.g., seasonal changes or "running out" cycles).
Future work could involve extending this deconfounder approach to more complex sequential models or exploring how to substitute other modules in the recommendation pipeline to handle different types of systemic bias beyond just frequency.
Summary updated