Towards More Relevant Product Search Ranking Via Large Language Models: An Empirical Study

Qi Liu, Atul Kumar Singh, Jingbo Liu, Cun Mu, Zheng Yan

Abstract

Training Learning-to-Rank models for e-commerce product search ranking can be challenging due to the lack of a gold standard of ranking relevance. In this paper, we decompose ranking relevance into content-based and engagement-based aspects, and we propose to leverage Large Language Models (LLMs) for both label and feature generation in model training, primarily aiming to improve the model's predictive capability for content-based relevance. Additionally, we introduce different sigmoid transformations on the LLM outputs to polarize relevance scores in labeling, enhancing the model's ability to balance content-based and engagement-based relevances and thus prioritize highly relevant items overall. Comprehensive online tests and offline evaluations are also conducted for the proposed design. Our work sheds light on advanced strategies for integrating LLMs into e-commerce product search ranking model training, offering a pathway to more effective and balanced models with improved ranking relevance.

Introduction

In the world of e-commerce, "relevance" is a moving target. A search for "blue running shoes" presents a conflict between engagement-based relevance (popularity, clicks) and content-based relevance (semantic match to description). Traditional Learning-to-Rank (LTR) models often struggle because there is no "gold standard" for relevance. Relying solely on user clicks can lead to popularity bias, while relying purely on text matching might ignore what customers actually want to buy. This paper addresses the challenge of integrating deep semantic understanding from LLMs into a production ranking system that must balance these two competing forces.

Methodology

The researchers proposed a multi-pronged strategy to integrate LLMs into the ranking pipeline:

  1. Decomposed Labeling: Training labels are defined as a product of two parts: $y = \sigma(C) \cdot E$, where $C$ is the content relevance and $E$ is the engagement score (clicks, add-to-carts).
  2. LLMs as Labelers: A Mistral 7B model, fine-tuned on human-evaluated data, was used to predict the content relevance score ($C$) offline.
  3. LLMs as Feature Generators: For real-time inference, a smaller, faster BERT-based cross-encoder generates content features, ensuring the model has semantic signals for ranking decisions.
  4. The Polarization Trick: This key innovation involves passing LLM content scores through a sigmoid transformation. By adjusting the steepness ($\alpha$) and the threshold ($\beta$), the scores are "polarized"—pushing mediocre matches toward zero and high-quality matches toward one.
Sigmoid transformations used to polarize content relevance scores across different regions.

Results and Discussion

The study found that simply adding LLM-based features or labels independently yielded minimal gains. However, using them together created a massive boost, as the LLM labels "teach" the model that the semantic features are the most important signals to follow.

Impact of Polarization

By shifting the sigmoid center ($\beta$), the researchers could tune the model's "strictness." As shown in the distribution shifts below, different configurations push the relevance scores toward the extremes, effectively acting as a filter for high-quality content.

Distribution of relevance scores for different polarization variants, showing how varying parameters shifts the model's focus.
  • Rigorous Variant ($\sigma_r LX$): Using a higher threshold ($\beta=0.7$), this model achieved a +3.96% lift in NDCG@10, significantly improving content relevance at the cost of a slight dip in engagement.
  • Relaxed Variant ($\sigma_l LX$): With a lower threshold ($\beta=0.3$), this variant increased Add-to-Cart (ATC@40) rates by +1.04%, prioritizing engagement while slightly compromising on strict content matching.
  • Balanced Variant ($\sigma_c LX$): This configuration achieved significant relevance gains while remaining neutral on business metrics like Gross Merchandise Volume (GMV).

SHAP value analysis confirmed that this polarization technique forced the model to rely more heavily on semantic features during inference, rather than just falling back on popularity metrics.

Conclusion

This research demonstrates that LLMs can effectively bridge the gap between semantic relevance and user engagement in e-commerce search. While there is an inherent trade-off between content relevance and engagement, the use of polarizing sigmoid transformations allows for fine-tuned control over this balance. Future work will focus on dynamically adjusting these parameters based on query types and optimizing inference latency for large-scale production environments.

Summary updated

Abstract

Training Learning-to-Rank models for e-commerce product search ranking can be challenging due to the lack of a gold standard of ranking relevance. In this paper, we decompose ranking relevance into content-based and engagement-based aspects, and we propose to leverage Large Language Models (LLMs) for both label and feature generation in model training, primarily aiming to improve the model's predictive capability for content-based relevance. Additionally, we introduce different sigmoid transformations on the LLM outputs to polarize relevance scores in labeling, enhancing the model's ability to balance content-based and engagement-based relevances and thus prioritize highly relevant items overall. Comprehensive online tests and offline evaluations are also conducted for the proposed design. Our work sheds light on advanced strategies for integrating LLMs into e-commerce product search ranking model training, offering a pathway to more effective and balanced models with improved ranking relevance.
BibTeX
@misc{liu2024towards,
  author = {Qi Liu and Atul Kumar Singh and Jingbo Liu and Cun Mu and Zheng Yan},
  title = {{Towards More Relevant Product Search Ranking Via Large Language Models: An Empirical Study}},
  year = {2024},
  eprint = {2409.17460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.17460},
}
Figure