TL;DR
This paper tackles the challenge of classifying e-commerce search queries, particularly those that are infrequent ("long-tail"). They propose a novel method that leverages contrastive learning to improve representations of these long-tail queries by transferring knowledge from more frequent, lexically similar queries. This approach significantly boosts classification performance for these historically difficult-to-handle queries.
What Problem Does This Paper Address?
In e-commerce, accurately classifying search queries into product categories is crucial for retrieving relevant products. While deep learning methods have shown success, they struggle with "long-tail" queries – those that appear infrequently in user search logs. This infrequency leads to sparse and noisy training data, resulting in poor classification performance for these queries compared to common, "frequent" ones. Furthermore, subtle lexical differences in short, ambiguous e-commerce queries can drastically alter their intent, making classification even harder, especially for long-tail queries where distinguishing such nuances is challenging due to limited examples.
Methods
The authors propose a two-module approach:
- Main Module: A standard BERT-based text classification model that takes a query and its potential category description as input to predict their relevance. This module is used for inference.
- Auxiliary Module: This module, also BERT-based and sharing parameters with the main module, is designed to enhance query representations. It uses contrastive learning to align the representations of a long-tail query with those of lexically similar, frequent "variant" queries. These variant queries act as anchors. The contrastive loss pulls together representations of queries and categories that are semantically aligned and pushes apart those that are not. This auxiliary module is trained but not used during inference, thus not adding computational overhead.
Key to their method is the Variant Query Selection process. They identify frequent queries that are lexically similar to a given long-tail query. These similar-but-potentially-different-intent queries serve as "hard negative examples" in the contrastive learning framework. The similarity is measured using a weighted token similarity based on TF-IDF scores.
The overall loss function combines the main module's classification loss with the auxiliary module's contrastive loss, re-weighted by parameters $\lambda_A$ and $\lambda_C$.
Datasets
- AliExpress Click Data: Used for training. This dataset consists of approximately 33.4 million (query, category) pairs, derived from 3 months of user click logs. Queries with daily average page views below 100 are considered long-tail.
- Public Dataset (CIKM Cup 2016): Used for evaluation. This dataset contains 500,000 (query, category) pairs from a personalized e-commerce search challenge.
Key Insights and Ideas
- Leveraging "Hard Negatives": The core idea of using lexically similar but potentially category-different frequent queries as anchors in contrastive learning is a clever way to expose the model to subtle distinctions that are crucial for long-tail query classification.
- Knowledge Transfer from Frequent to Long-Tail: The method effectively transfers reliable supervised signals from abundant click data of frequent queries to improve the understanding of infrequent long-tail queries.
- Contrastive Learning for Representation Alignment: The application of contrastive loss is shown to be effective in aligning query and category representations in a way that better captures semantic relationships, especially for ambiguous queries.
- Auxiliary Module for Enhanced Representations: The auxiliary module, trained with contrastive loss, demonstrably improves the quality of query representations, leading to better downstream classification.
Key Results
- Significant Performance Improvement: The proposed method achieved statistically significant improvements across multiple metrics (AUC, AP, Precision, Recall, F1 Score) compared to baseline methods like a standard BERT model and prior work ("AC(LSTM)", "AC(BERT)"). For instance, on AliExpress data, it showed a +1.92% improvement in AUC and +2.48% in AP over the "Base" model.
- Effectiveness on Long-Tail Queries: The performance gains were most pronounced on the "Long-tail Tail" query set, indicating the method's particular strength in handling the most infrequent queries.
- Visualization Confirms Better Representations: UMAP visualizations showed that the proposed method clusters long-tail queries by category much more effectively than the baseline, demonstrating that the learned representations better capture category semantics.
- Online Evaluation Success: The method led to a +0.63% increase in Click-Through Rate (CTR) and a +1.06% increase in Revenue Per Mille (RPM) in an online A/B test on AliExpress, highlighting its practical impact.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the "Proposed-Entire Query" ablation test showed that using only frequent queries as variant candidates is better than using all queries, suggesting that the quality of anchors matters.
- The visualization of query clusters still showed some overlaps, particularly for queries naturally related to multiple categories (e.g., "T-shirt (Men)" vs. "T-shirt (Women)").
- Future Work:
- Generalizing the approach to more situations in e-commerce, such as multi-language query classification.
- Extending the idea to other related tasks like product retrieval or relevance classification.
Summary updated