Combination of Diverse Ranking Models for Personalized Expedia Hotel Searches

Xudong Liu, Bing Xu, Yuyu Zhang, Qiang Yan, Liang Pang, Qiang Li, Hanxiao Sun, Bin WangarXiv 2013 · OpenReview

Abstract

This paper presents the 5th-place solution for the 2013 ICDM Challenge, which focused on personalizing hotel search rankings for Expedia. The approach leverages an ensemble of highly diverse models, including Gradient Boosting Machines, Random Forests, Factorization Machines, and Deep Learning architectures. By combining sophisticated feature engineering—particularly listwise features—with robust ensemble strategies like Z-score normalization and listwise stacking, the authors demonstrate significant improvements over single-model baselines in a highly imbalanced e-commerce ranking environment.

Introduction

Learning to Rank (LTR) is a critical component of modern e-commerce, where the goal is to present users with items most likely to result in a conversion. In the context of Expedia's hotel search, this task is complicated by extreme class imbalance (only 4.4% positive outcomes), the need to integrate diverse data types (user history, hotel attributes, and competitive pricing), and the inherent listwise nature of the ranking problem. This paper details a multi-stage pipeline designed to address these challenges through model diversity and contextual feature engineering.

Methodology

Feature Engineering and Data Preparation

To capture the context of a search query, the authors developed "listwise features," such as the rank of a hotel's price relative to others in the same search result. They also utilized composite features and "bucketing" to transform continuous variables into discrete bins, facilitating better pattern recognition in certain models. To handle the severe class imbalance, training data for tree-based and deep learning models was manually balanced to a 1:1 ratio of positive to negative samples.

Diverse Base Models

The system employs a wide array of algorithms categorized by their learning approach:

  • Pointwise: Logistic Regression with weighted cross-entropy and Factorization Machines.
  • Pairwise: FTRL-Proximal and SVMRank.
  • Tree-Based: Random Forests (partitioned by country), Gradient Boosting Machines (GBM), and Extreme Randomized Trees (ERT).
  • Listwise: LambdaMART.
  • Deep Learning: A specialized architecture involving Denoising Autoencoders and Maxout Networks.

Ensemble Architecture

The core of the solution is the integration of these diverse models. The authors utilized a multi-layered ensemble approach to aggregate predictions.

Overall ensemble architecture showing the flow from data infrastructure to final outputs via diverse base models and ensemble methods.

To effectively combine scores from disparate models (e.g., a Factorization Machine vs. a Neural Network), Z-score normalization was applied within each query. Final rankings were generated using GBM stacking and a top-level LambdaMART ensemble that treated base model outputs as input features.

Deep Learning Component

While decision trees often dominate tabular data tasks, the authors explored deep learning to capture non-linear interactions.

The deep model utilized stacked Denoising Autoencoders for unsupervised feature learning, followed by Maxout layers to improve gradient flow and prevent saturation. However, the authors noted that these models were particularly sensitive to the tabular data's imbalance and required careful tuning.

Results and Evaluation

The models were evaluated using the Expedia Hotel Search Dataset, comprising approximately 9.9 million training points and 6.6 million test points. The primary metric was Normalized Discounted Cumulative Gain (NDCG@38).

The final ensemble achieved an NDCG@38 score of 0.53102, securing 5th place on the private leaderboard. In comparison, a standalone GBM model reached 0.52477. This improvement underscores the value of model diversity; the combination of over 30 distinct models successfully captured different nuances of user behavior and hotel relevance that no single algorithm could master alone.

Discussion and Key Insights

A primary takeaway is the importance of contextual features. Raw price data was found to be less predictive than the relative price rank within a search result. Furthermore, Z-score normalization proved to be a "great equalizer," allowing the ensemble to benefit from models with wildly different output scales.

Despite the promise of the deep learning architecture, it struggled to outperform tree-based ensembles on this specific dataset. The authors attribute this to the difficulty of training deep networks on highly imbalanced tabular data and the lack of sufficient data for fine-grained subsets (e.g., specific countries).

Conclusion

The success of this approach in the ICDM Challenge 2013 demonstrates that for complex e-commerce ranking tasks, a "divide and conquer" strategy using diverse models is highly effective. Future work could focus on better normalization techniques for deep learning on tabular data and more exhaustive feature selection for listwise ranking biases.

Summary updated

Abstract

The ICDM Challenge 2013 is to apply machine learning to the problem of hotel ranking, aiming to maximize purchases according to given hotel characteristics, location attractiveness of hotels, user's aggregated purchase history and competitive online travel agency information for each potential hotel choice. This paper describes the solution of team "binghsu & MLRush & BrickMover". We conduct simple feature engineering work and train different models by each individual team member. Afterwards, we use listwise ensemble method to combine each model's output. Besides describing effective model and features, we will discuss about the lessons we learned while using deep learning in this competition.
BibTeX
@misc{liu2013combination,
  author = {Xudong Liu and Bing Xu and Yuyu Zhang and Qiang Yan and Liang Pang and Qiang Li and Hanxiao Sun and Bin Wang},
  title = {{Combination of Diverse Ranking Models for Personalized Expedia Hotel Searches}},
  year = {2013},
  eprint = {1311.7679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1311.7679},
}
Figure