Abstract
Personalized search has been a hot research topic for many years and has been widely used in e-commerce. This paper describes our solution to tackle the challenge of personalized e-commerce search at CIKM Cup 2016. The goal of this competition is to predict search relevance and re-rank the result items in SERP according to the personalized search, browsing and purchasing preferences. Based on a detailed analysis of the provided data, we extract three different types of features, i.e., statistic features, query-item features and session features. Different models are used on these features, including logistic regression, gradient boosted decision trees, rank svm and a novel deep match model. With the blending of multiple models, a stacking ensemble model is built to integrate the output of individual models and produce a more accurate prediction result. Based on these efforts, our solution won the champion of the competition on all the evaluation metrics.
Introduction
In the competitive landscape of e-commerce, delivering highly relevant search results is critical for conversion. However, relevance is inherently subjective, shaped by individual user history and intent. The CIKM Cup 2016 challenge focused on this problem, requiring participants to predict search relevance and re-rank items in two distinct scenarios: Query-full (standard keyword-based search) and Query-less (browsing scenarios where users click categories without specific search terms).
Key hurdles included processing massive anonymized datasets, managing a high volume of anonymous users without long-term history, and bridging the semantic gap between concise search queries and verbose product descriptions.
Methodology
The winning solution treats the task as a Learning to Rank (LTR) problem, employing a sophisticated multi-stage pipeline.
Feature Engineering
The core of the approach lies in three distinct feature categories:
- Global Statistics: These include historical click-through rates (CTR), conversion rates (CVR), and a "cost performance ratio" (behavior counts normalized by price).
- Query-Item Features: To address the low word overlap (only 1.8%) between queries and items, the team utilized word-level Cartesian products and vector-level embeddings.
- Session Features: These track short-term user behavior, such as repeated views or clicks on the same product within a single session.
Deep Match Model (DMM)
To overcome the limitations of traditional keyword matching (like BM25), the authors developed a supervised Deep Match Model. This model utilizes Word2vec-style embeddings, trained on user feedback (clicks and purchases), to project queries and items into a shared semantic vector space.
Stacking Ensemble
The final predictions are generated by a two-layer stacking architecture. The first layer consists of diverse base learners, including Logistic Regression (LR), RankSVM, and the DMM. Their outputs are then fed into a Gradient Boosted Decision Tree (GBDT) meta-learner, which learns to optimally combine these scores into a final prediction.
Dataset and Analysis
The solution was developed using the DIGINETICA dataset, which contains:
- Search/Browsing Logs: ~53k query-full and ~870k query-less queries.
- Interaction Data: Over 1.1 million click logs and 1.2 million view logs, contrasted against only ~18k purchase records, indicating high data sparsity for conversions.
- User Profiles: A mix of ~233k registered users and ~333k anonymous users.
Key Insights
- Value Perception: While raw price is a weak feature, the "cost performance" feature (Behavior Count / Price) proved highly effective at capturing perceived value.
- Session Persistence: Over 20% of clicks in a session's final query were "repeats" of previously viewed products, highlighting the importance of session-based personalization.
- Semantic vs. Keyword Matching: The DMM demonstrated that supervised semantic embeddings are far more effective than simple text overlap for messy e-commerce data.
- Algorithmic Diversity: The ensemble's success stemmed from combining models with different strengths; LR handled complex feature combinations well, while GBDT captured non-linear interactions.
Results
The ensemble approach secured the champion title in the CIKM Cup 2016 across all evaluation metrics.
- Validation Performance: The final ensemble achieved an overall NDCG (Normalized Discounted Cumulative Gain) of 0.4238.
- Component Analysis: Logistic Regression was the strongest individual model (0.4175 NDCG), but the stacking method provided the necessary margin for victory.
- Scenario Weighting: The final evaluation was weighted 80% toward query-less scenarios and 20% toward query-full, reflecting the prevalence and difficulty of browsing behavior.
Conclusion
The success of this approach underscores the power of combining deep semantic modeling with robust ensemble techniques. Future work could involve integrating collaborative filtering to further enhance personalization in query-less scenarios and exploring more complex deep learning architectures as data volume increases.
Summary updated