Abstract
Showing items that do not match search query intent degrades customer experience in e-commerce. These mismatches result from counterfactual biases of the ranking algorithms toward noisy behavioral signals such as clicks and purchases in the search logs. Mitigating the problem requires a large labeled dataset, which is expensive and time-consuming to obtain. In this paper, we develop a deep, end-to-end model that learns to effectively classify mismatches and to generate hard mismatched examples to improve the classifier. We train the model end-to-end by introducing a latent variable into the cross-entropy loss that alternates between using the real and generated samples. This not only makes the classifier more robust but also boosts the overall ranking performance. Our model achieves a relative gain compared to baselines by over 26% in F-score, and over 17% in Area Under PR curve. On live search traffic, our model gains significant improvement in multiple countries.
Introduction
In e-commerce, presenting irrelevant products to customers—known as mismatches—directly harms the user experience. Identifying these mismatches is challenging due to two primary factors. First, search engines often rely on noisy behavioral data; for instance, a user might search for a "phone" but end up purchasing a "phone case," leading the algorithm to incorrectly associate the two. Second, while "easy" mismatches (e.g., searching for shoes and seeing a telescope) are simple to filter, "hard" mismatches (e.g., searching for "iPhone X" and seeing an "iPhone X charger") are lexically similar but semantically distinct. Manually labeling a sufficient volume of these hard examples is prohibitively expensive.
The QUARTS Framework
To address these challenges, we propose QUARTS (QUery-based Adversarial learning for Robust Textual Search). QUARTS is an end-to-end framework that combines a classifier with an adversarial query generator to synthesize hard negative examples.
The framework consists of three core components:
- LSTM Classifier: Utilizes a word-by-word attention mechanism to evaluate the relationship between a query and a product title.
- Variational Query Generator (VED): Acts as an adversary. Given a valid query-item pair, it generates a modified query that is realistic but mismatched with the item (e.g., changing "running shoe" to "insoles for running shoes").
- State Combiner: Instead of generating discrete text, which complicates optimization, the model merges the continuous vector representations of real and generated queries. A latent variable $z$ serves as a switch, allowing the model to alternate between real and adversarial data during training.
Experimental Setup
The model was evaluated using a massive dataset from a major e-commerce platform:
- Human-Labeled Data: 3.2 million query-item pairs.
- Search Log Data: 3 million pairs identified as matches based on purchase frequency.
- Evaluation: A test set of 100,000 labeled pairs was used to measure Area Under PR curve (APR) and F1-score.
Results and Analysis
QUARTS demonstrated significant improvements over traditional and deep learning baselines, including BERT-based models. Offline, it achieved a 17.15% relative gain in APR and a 26.06% gain in F-score. Online A/B testing showed a 12.2% and 5.75% increase in key metrics across different regions.
A critical factor in this success is the model's attention mechanism. By visualizing the attention weights, we can see the model focusing on "modifier" words—such as "case" or "protector"—that fundamentally alter the query's intent.
Human evaluation confirmed the quality of the generator, with 82% of synthesized queries deemed realistic modifications of original intents.
Conclusion
QUARTS provides a robust solution for e-commerce search by automating the generation of hard training examples. By operating on hidden representations, it avoids the complexities of reinforcement learning typically associated with adversarial text generation. Future work will explore expanding the generator to include diverse mismatch types, such as those based on price points or complementary goods, and applying the framework to non-e-commerce domains.
Summary updated