Abstract
This paper presents the 5th-place solution for the 2013 ICDM Challenge, which focused on personalizing hotel search rankings for Expedia. The approach leverages an ensemble of highly diverse models, including Gradient Boosting Machines, Random Forests, Factorization Machines, and Deep Learning architectures. By combining sophisticated feature engineering—particularly listwise features—with robust ensemble strategies like Z-score normalization and listwise stacking, the authors demonstrate significant improvements over single-model baselines in a highly imbalanced e-commerce ranking environment.
Introduction
Learning to Rank (LTR) is a critical component of modern e-commerce, where the goal is to present users with items most likely to result in a conversion. In the context of Expedia's hotel search, this task is complicated by extreme class imbalance (only 4.4% positive outcomes), the need to integrate diverse data types (user history, hotel attributes, and competitive pricing), and the inherent listwise nature of the ranking problem. This paper details a multi-stage pipeline designed to address these challenges through model diversity and contextual feature engineering.
Methodology
Feature Engineering and Data Preparation
To capture the context of a search query, the authors developed "listwise features," such as the rank of a hotel's price relative to others in the same search result. They also utilized composite features and "bucketing" to transform continuous variables into discrete bins, facilitating better pattern recognition in certain models. To handle the severe class imbalance, training data for tree-based and deep learning models was manually balanced to a 1:1 ratio of positive to negative samples.
Diverse Base Models
The system employs a wide array of algorithms categorized by their learning approach:
- Pointwise: Logistic Regression with weighted cross-entropy and Factorization Machines.
- Pairwise: FTRL-Proximal and SVMRank.
- Tree-Based: Random Forests (partitioned by country), Gradient Boosting Machines (GBM), and Extreme Randomized Trees (ERT).
- Listwise: LambdaMART.
- Deep Learning: A specialized architecture involving Denoising Autoencoders and Maxout Networks.
Ensemble Architecture
The core of the solution is the integration of these diverse models. The authors utilized a multi-layered ensemble approach to aggregate predictions.
To effectively combine scores from disparate models (e.g., a Factorization Machine vs. a Neural Network), Z-score normalization was applied within each query. Final rankings were generated using GBM stacking and a top-level LambdaMART ensemble that treated base model outputs as input features.
Deep Learning Component
While decision trees often dominate tabular data tasks, the authors explored deep learning to capture non-linear interactions.
The deep model utilized stacked Denoising Autoencoders for unsupervised feature learning, followed by Maxout layers to improve gradient flow and prevent saturation. However, the authors noted that these models were particularly sensitive to the tabular data's imbalance and required careful tuning.
Results and Evaluation
The models were evaluated using the Expedia Hotel Search Dataset, comprising approximately 9.9 million training points and 6.6 million test points. The primary metric was Normalized Discounted Cumulative Gain (NDCG@38).
The final ensemble achieved an NDCG@38 score of 0.53102, securing 5th place on the private leaderboard. In comparison, a standalone GBM model reached 0.52477. This improvement underscores the value of model diversity; the combination of over 30 distinct models successfully captured different nuances of user behavior and hotel relevance that no single algorithm could master alone.
Discussion and Key Insights
A primary takeaway is the importance of contextual features. Raw price data was found to be less predictive than the relative price rank within a search result. Furthermore, Z-score normalization proved to be a "great equalizer," allowing the ensemble to benefit from models with wildly different output scales.
Despite the promise of the deep learning architecture, it struggled to outperform tree-based ensembles on this specific dataset. The authors attribute this to the difficulty of training deep networks on highly imbalanced tabular data and the lack of sufficient data for fine-grained subsets (e.g., specific countries).
Conclusion
The success of this approach in the ICDM Challenge 2013 demonstrates that for complex e-commerce ranking tasks, a "divide and conquer" strategy using diverse models is highly effective. Future work could focus on better normalization techniques for deep learning on tabular data and more exhaustive feature selection for listwise ranking biases.
Summary updated