RecSys Challenge 2016: job recommendations based on preselection of offers and gradient boosting

Andrzej Pacuk, Piotr Sankowski, Karol Wegrzycki, Adam Witkowski, Piotr Wygocki

TL;DR

The Mim-Solution team secured 2nd place in the RecSys Challenge 2016 by deploying a robust two-stage pipeline for job recommendations on the XING platform. Their approach combined heuristic-based candidate preselection to handle massive scale with a high-performance Gradient Boosting Decision Tree (XGBoost) model to rank those candidates based on the probability of user interaction.

What Problem Does This Paper Address?

Recommender systems in the wild face a "needle in a haystack" problem. For the XING platform, the task was to recommend 30 job offers to a target user that they would likely click, bookmark, or reply to.

The challenge is twofold:

  1. Scale: With 150,000 target users and over 300,000 active job items, calculating a score for every possible user-item pair (45 billion combinations) is computationally infeasible for most teams.
  2. Sparsity and Anonymization: The data was anonymized into numerical values, making intuitive feature engineering difficult, and the "ground truth" (what users actually liked) was only available for a tiny fraction of the total impressions.

Methods

The authors utilized a classic "Retrieve and Rank" architecture:

  1. Candidate Selection (Retrieval): To make the problem tractable, they narrowed the pool from 300,000+ items to roughly 300 "promising" candidates per user. They used nine different heuristics, including:
  2. Items the user previously interacted with.
  3. Items liked by "similar" users (using Jaccard similarity on interaction history).
  4. Items with high tag or title similarity to the user's history.
  5. Global "most popular" items.
  6. Learning the Probabilities (Ranking): Once the candidates were selected, they used XGBoost (Gradient Boosting Decision Trees) to predict the probability $P[i \in G(u)]$ that a user $u$ would interact with item $i$.
  7. Objective Function: They optimized for logloss rather than the competition's complex scoring metric directly, finding that predicting raw interaction probability was more stable.
  8. Feature Engineering: They generated 273 features, ranging from "event-based" stats (e.g., what percentage of items a user liked had a specific career level) to "temporal" features (e.g., seconds since the last user activity).
  9. Blending: The final submission was an arithmetic mean of their best-performing models' predicted probabilities.

Datasets

The study used the RecSys Challenge 2016 dataset provided by XING. It is a massive, anonymized dataset reflecting three months of platform activity:

  • Impressions: ~1.08 Billion (items shown to users).
  • Interactions: ~8.8 Million (clicks, bookmarks, replies, deletes).
  • Users: 1.5 Million (150,000 target users).
  • Items: ~1.3 Million (327,003 active during the test period).

Key Insights and Ideas

  • The Preselection Bottleneck: The authors discovered that their candidate selection phase only covered 37% of the actual ground truth. This means that even with a "perfect" ranking model, they could only ever achieve 37% of the maximum possible score. This highlights that in large-scale RecSys, the retrieval layer is often the most significant constraint on performance.
  • Popularity is King (Mostly): "Event-based" features and "Item global popularity" were by far the most important features for the XGBoost model. In job hunting, trending or high-quality posts tend to attract engagement across the board.
  • Simplicity in Optimization: While the competition had a very specific, weighted evaluation metric (rewarding precision at different ranks), the authors found that simply training the model to minimize logloss (binary classification of "will they click?") was highly effective.
  • The "Delete" Signal: They specifically excluded items that a user had previously deleted, treating negative feedback as a hard constraint.

Key Results

  • Competition Standing: The approach ranked 2nd overall out of many international teams.
  • Leaderboard Scores: They achieved a score of 675,985.03 on the public leaderboard and 2,035,964.16 on the private leaderboard.
  • Model Efficiency: By limiting candidates to ~300 per user, they reduced the workload from billions of pairs to about 43 million, making it possible to train on a single 12-core server with 64GB of RAM.

Limitations and Future Work

  • Candidate Coverage: As noted, the 37% coverage of the ground truth was a major limitation. The authors suggest that expanding the candidate set or using more sophisticated retrieval methods (like embedding-based nearest neighbor search) could yield massive improvements.
  • Anonymization Hurdles: The authors attempted to use similarity based on user/item properties (region, industry), but these didn't improve the score significantly. They suspect the anonymization process stripped away the nuances that would have made these features useful.
  • Training Set Size: Post-deadline analysis showed that simply increasing the number of negative samples in the training set (using 1/4 of available negatives instead of a smaller subset) boosted the score significantly, suggesting the model was still data-hungry.

Summary updated

Abstract

We present the Mim-Solution's approach to the RecSys Challenge 2016, which ranked 2nd. The goal of the competition was to prepare job recommendations for the users of the website Xing.com. Our two phase algorithm consists of candidate selection followed by the candidate ranking. We ranked the candidates by the predicted probability that the user will positively interact with the job offer. We have used Gradient Boosting Decision Trees as the regression tool.
BibTeX
@inproceedings{pacuk2016recsys,
  author = {Andrzej Pacuk and Piotr Sankowski and Karol Wegrzycki and Adam Witkowski and Piotr Wygocki},
  title = {{RecSys Challenge 2016: job recommendations based on preselection of offers and gradient boosting}},
  booktitle = {RecSys Challenge},
  year = {2016},
  url = {https://openreview.net/forum?id=oGIIu57uYA},
}

Loading related content...

Figure