Long or Short or Both? An Exploration on Lookback Time Windows of Behavioral Features in Product Search Ranking

Qi Liu, Atul Kumar Singh, Jingbo Liu, Cun Mu, Zheng Yan, Jan Fog PedersenarXiv 2024 · DOI · arXiv

TL;DR

Researchers at Walmart Global Tech investigated the optimal "lookback window" for behavioral features (like click and purchase rates) in product search ranking. They discovered that while long-term data provides stability and short-term data captures trends, simply combining them can actually hurt performance; instead, they developed a hybrid model that uses query-level category signals (e.g., "Food" vs. "Fashion") to intelligently weight these different time horizons, resulting in significant lifts in real-world eCommerce metrics.

What Problem Does This Paper Address?

In eCommerce, ranking models rely heavily on behavioral signals: how often users click on, add to cart (ATC), or buy a specific product for a specific query. A major headache for engineers is deciding how much historical data to aggregate.

If you use a long lookback window (e.g., 2 years), your data is robust and less noisy, but it’s "stale"—it might keep ranking 2023 calendars at the top in late 2024. If you use a short lookback window (e.g., 1 month), you capture the latest trends and new product launches, but the data is "sparse" and "noisy"—a single random purchase can skew the ranking of an irrelevant product.

The authors sought to move beyond the binary choice of "long vs. short" to find a principled way to integrate both without the signals diluting each other.

Methods

The researchers compared several iterations of ranking models using XGBoost within a Learning-to-Rank (LTR) framework.

  1. Feature Smoothing: To handle the "cold start" problem (new products with zero data), they used a Beta-Binomial Bayesian model. This smooths raw counts with a "prior" distribution, ensuring that a product with 1 click out of 1 view isn't automatically ranked higher than one with 900 clicks out of 1,000 views.
  2. Model Variations:
    • Baseline: 2-year lookback only.
  3. Model A: 1-month lookback only.
  4. Model B: A "vanilla" hybrid using both 2-year and 1-month features.
  5. Model C (The Winner): Both features plus query-level vertical signals. These are one-hot encoded categories (e.g., Food, Fashion, Electronics) predicted by an upstream query-understanding model.
  6. Evaluation: They used interleaving tests (a sensitive way to compare two ranking algorithms by mixing their results) and traditional A/B tests on live Walmart.com traffic.

Datasets

The study was conducted using live production data from Walmart.com.

  • Timeframes: 730 days (long) vs. 30 days (short).
  • Verticals: Queries were categorized into six business niches: Food, Consumables, Home, Hardlines, Fashion, and ETS (Electronics, Toys, and Seasonal).
  • Scale: The tests were performed on a "substantial volume" of online customer traffic to ensure statistical significance.

Key Insights and Ideas

  • The "Vanilla" Hybrid Failure: Surprisingly, Model B (simply adding both features) performed worse than the baseline in several categories. The authors suggest this is because the model lacked the flexibility to know when to trust which feature. Adding short-term noise to stable categories (like "Milk") or long-term staleness to trendy categories (like "New Year's Eve Decor") diluted the overall ranking quality.
  • Vertical-Specific Sensitivity:
    • Food and Consumables benefit from long-term data because shopping patterns are stable and enduring.
  • General Merchandise (Fashion, Electronics) benefits significantly from short-term data because inventory and trends change rapidly.
  • The "Gatekeeper" Effect: By adding query-level vertical signals, the XGBoost trees learned to use the vertical as a "router." Analysis of the tree nodes showed that when the query was "Fashion," the model naturally prioritized 1-month features at the top of the decision tree. For "Consumables," it prioritized 2-year features.

Key Results

The hybrid model guided by vertical signals (Model C) outperformed all others:

  • Overall Engagement: A statistically significant +0.22% lift in customer engagement across all searches.
  • Marketplace GMV: A +0.64% increase in Gross Merchandise Value for marketplace products, suggesting the short-term windows helped surface newer marketplace items faster (solving the cold-start problem).
  • User Experience: Significant improvements in ATC@10 (Add-to-Carts in the top 10 results) and a -0.16% reduction in Session Abandonment.
  • Qualitative Success: In a real-world example from December 2023, the new model successfully demoted "2023" themed items and promoted "2024" items, whereas the baseline model was still stuck on the previous year's data.

Limitations and Future Work

  • Granularity: The current model uses broad "verticals." The authors plan to introduce more granular signals, such as specific product types or categorical levels, to provide even more nuanced guidance to the behavioral features.
  • More Windows: The study only compared 1-month vs. 2-years. Future work will explore additional windows, such as 1-week (for hyper-recent trends) and 1-year (to capture seasonality better).
  • Feature Scope: They intend to expand the scope of behavioral features beyond just click, ATC, and order rates to include more complex user interaction signals.

Summary updated

Abstract

Customer shopping behavioral features are core to product search ranking models in eCommerce. In this paper, we investigate the effect of lookback time windows when aggregating these features at the (query, product) level over history. By studying the pros and cons of using long and short time windows, we propose a novel approach to integrating these historical behavioral features of different time windows. In particular, we address the criticality of using query-level vertical signals in ranking models to effectively aggregate all information from different behavioral features. Anecdotal evidence for the proposed approach is also provided using live product search traffic on Walmart.com.
BibTeX
@misc{liu2024long,
  author = {Qi Liu and Atul Kumar Singh and Jingbo Liu and Cun Mu and Zheng Yan and Jan Fog Pedersen},
  title = {{Long or Short or Both? An Exploration on Lookback Time Windows of Behavioral Features in Product Search Ranking}},
  year = {2024},
  eprint = {2409.17456},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.17456},
}
Figure