RecSys Challenge 2016: job recommendations based on preselection of offers and gradient boosting

Andrzej Pacuk, Piotr Sankowski, Karol Wegrzycki, Adam Witkowski, Piotr Wygocki

Abstract

We present the Mim-Solution's approach to the RecSys Challenge 2016, which ranked 2nd. The goal of the competition was to prepare job recommendations for the users of the website Xing.com. Our two phase algorithm consists of candidate selection followed by the candidate ranking. We ranked the candidates by the predicted probability that the user will positively interact with the job offer. We have used Gradient Boosting Decision Trees as the regression tool.

Introduction

The Mim-Solution team secured 2nd place in the RecSys Challenge 2016 by deploying a robust two-stage pipeline for job recommendations on the XING platform. Recommender systems in production environments often face a "needle in a haystack" problem. For XING, the task was to recommend 30 job offers to a target user that they would likely click, bookmark, or reply to.

The challenge was primarily defined by scale and data sparsity. With 150,000 target users and over 300,000 active job items, calculating a score for every possible user-item pair (45 billion combinations) is computationally infeasible. Furthermore, the data was anonymized, making intuitive feature engineering difficult, and positive interaction data was sparse relative to the total number of impressions.

Methodology

The authors utilized a classic "Retrieve and Rank" architecture to manage the computational load while maintaining high recommendation quality.

The two-phase recommendation pipeline: candidate selection followed by probability-based ranking.

1. Candidate Selection (Retrieval)

To make the problem tractable, the pool was narrowed from 300,000+ items to roughly 300 "promising" candidates per user using nine heuristics, including:

  • Items the user previously interacted with.
  • Items liked by "similar" users (using Jaccard similarity on interaction history).
  • Items with high tag or title similarity to the user's history.
  • Global "most popular" items.

2. Learning the Probabilities (Ranking)

Once candidates were selected, XGBoost (Gradient Boosting Decision Trees) predicted the probability $P[i \in G(u)]$ that a user $u$ would interact with item $i$.

  • Objective Function: The team optimized for logloss rather than the competition's complex scoring metric directly, finding that predicting raw interaction probability was more stable.
  • Feature Engineering: They generated 273 features, ranging from "event-based" stats (e.g., what percentage of items a user liked had a specific career level) to "temporal" features (e.g., seconds since the last user activity).

3. Blending

The final submission was an arithmetic mean of the best-performing models' predicted probabilities.

Datasets

The study used the RecSys Challenge 2016 dataset provided by XING, reflecting three months of platform activity. The dataset included approximately 1.08 billion impressions and 8.8 million interactions (clicks, bookmarks, replies, deletes). The target group consisted of 150,000 users out of a total 1.5 million, and 327,003 active items out of 1.3 million.

Key Insights and Results

The approach ranked 2nd overall, achieving a private leaderboard score of 2,035,964.16. By limiting candidates to ~300 per user, the workload was reduced from billions of pairs to about 43 million, allowing training on a single 12-core server with 64GB of RAM.

Feature importance ranking for the XGBoost model, highlighting the dominance of event-based and popularity features.

Analysis of the model revealed several critical insights:

  • The Preselection Bottleneck: The candidate selection phase only covered 37% of the actual ground truth. This highlights that in large-scale systems, the retrieval layer is often the primary constraint on performance.
  • Feature Importance: "Event-based" features and "Item global popularity" were the most significant predictors. In job seeking, trending posts tend to attract broad engagement.
  • Optimization Simplicity: Minimizing logloss for binary classification proved highly effective despite the complex competition metric.
  • Negative Signals: Excluding previously deleted items as a hard constraint improved results.

Conclusion

The Mim-Solution approach demonstrates the efficacy of a two-stage pipeline in large-scale recommendation tasks. While successful, the 37% ground truth coverage in the retrieval phase suggests that future work should focus on more sophisticated candidate generation, such as embedding-based nearest neighbor search. Additionally, post-deadline analysis indicated that increasing the ratio of negative samples in the training set could further boost performance.

Summary updated

Abstract

We present the Mim-Solution's approach to the RecSys Challenge 2016, which ranked 2nd. The goal of the competition was to prepare job recommendations for the users of the website Xing.com. Our two phase algorithm consists of candidate selection followed by the candidate ranking. We ranked the candidates by the predicted probability that the user will positively interact with the job offer. We have used Gradient Boosting Decision Trees as the regression tool.
BibTeX
@inproceedings{pacuk2016recsys,
  author = {Andrzej Pacuk and Piotr Sankowski and Karol Wegrzycki and Adam Witkowski and Piotr Wygocki},
  title = {{RecSys Challenge 2016: job recommendations based on preselection of offers and gradient boosting}},
  booktitle = {RecSys Challenge},
  year = {2016},
  url = {https://openreview.net/forum?id=oGIIu57uYA},
}
Figure