TL;DR
The Mim-Solution team secured 2nd place in the RecSys Challenge 2016 by deploying a robust two-stage pipeline for job recommendations on the XING platform. Their approach combined heuristic-based candidate preselection to handle massive scale with a high-performance Gradient Boosting Decision Tree (XGBoost) model to rank those candidates based on the probability of user interaction.
What Problem Does This Paper Address?
Recommender systems in the wild face a "needle in a haystack" problem. For the XING platform, the task was to recommend 30 job offers to a target user that they would likely click, bookmark, or reply to.
The challenge is twofold:
- Scale: With 150,000 target users and over 300,000 active job items, calculating a score for every possible user-item pair (45 billion combinations) is computationally infeasible for most teams.
- Sparsity and Anonymization: The data was anonymized into numerical values, making intuitive feature engineering difficult, and the "ground truth" (what users actually liked) was only available for a tiny fraction of the total impressions.
Methods
The authors utilized a classic "Retrieve and Rank" architecture:
- Candidate Selection (Retrieval): To make the problem tractable, they narrowed the pool from 300,000+ items to roughly 300 "promising" candidates per user. They used nine different heuristics, including:
- Items the user previously interacted with.
- Items liked by "similar" users (using Jaccard similarity on interaction history).
- Items with high tag or title similarity to the user's history.
- Global "most popular" items.
- Learning the Probabilities (Ranking): Once the candidates were selected, they used XGBoost (Gradient Boosting Decision Trees) to predict the probability $P[i \in G(u)]$ that a user $u$ would interact with item $i$.
- Objective Function: They optimized for logloss rather than the competition's complex scoring metric directly, finding that predicting raw interaction probability was more stable.
- Feature Engineering: They generated 273 features, ranging from "event-based" stats (e.g., what percentage of items a user liked had a specific career level) to "temporal" features (e.g., seconds since the last user activity).
- Blending: The final submission was an arithmetic mean of their best-performing models' predicted probabilities.
Datasets
The study used the RecSys Challenge 2016 dataset provided by XING. It is a massive, anonymized dataset reflecting three months of platform activity:
- Impressions: ~1.08 Billion (items shown to users).
- Interactions: ~8.8 Million (clicks, bookmarks, replies, deletes).
- Users: 1.5 Million (150,000 target users).
- Items: ~1.3 Million (327,003 active during the test period).
Key Insights and Ideas
- The Preselection Bottleneck: The authors discovered that their candidate selection phase only covered 37% of the actual ground truth. This means that even with a "perfect" ranking model, they could only ever achieve 37% of the maximum possible score. This highlights that in large-scale RecSys, the retrieval layer is often the most significant constraint on performance.
- Popularity is King (Mostly): "Event-based" features and "Item global popularity" were by far the most important features for the XGBoost model. In job hunting, trending or high-quality posts tend to attract engagement across the board.
- Simplicity in Optimization: While the competition had a very specific, weighted evaluation metric (rewarding precision at different ranks), the authors found that simply training the model to minimize logloss (binary classification of "will they click?") was highly effective.
- The "Delete" Signal: They specifically excluded items that a user had previously deleted, treating negative feedback as a hard constraint.
Key Results
- Competition Standing: The approach ranked 2nd overall out of many international teams.
- Leaderboard Scores: They achieved a score of 675,985.03 on the public leaderboard and 2,035,964.16 on the private leaderboard.
- Model Efficiency: By limiting candidates to ~300 per user, they reduced the workload from billions of pairs to about 43 million, making it possible to train on a single 12-core server with 64GB of RAM.
Limitations and Future Work
- Candidate Coverage: As noted, the 37% coverage of the ground truth was a major limitation. The authors suggest that expanding the candidate set or using more sophisticated retrieval methods (like embedding-based nearest neighbor search) could yield massive improvements.
- Anonymization Hurdles: The authors attempted to use similarity based on user/item properties (region, industry), but these didn't improve the score significantly. They suspect the anonymization process stripped away the nuances that would have made these features useful.
- Training Set Size: Post-deadline analysis showed that simply increasing the number of negative samples in the training set (using 1/4 of available negatives instead of a smaller subset) boosted the score significantly, suggesting the model was still data-hungry.
Summary updated