Abstract
In the competitive landscape of large-scale recommender systems, the choice between traditional Gradient Boosted Decision Trees (GBDTs) and modern Neural Networks is a critical architectural decision. This paper presents a comparative study conducted at ShareChat, evaluating the performance of Multi-gate Mixture of Experts (MMoE) neural rankers against Catboost GBDTs. Utilizing a production-scale dataset of 2 billion training points, we analyze accuracy, scalability, and hardware efficiency. Our findings indicate that while GBDTs remain competitive at smaller scales, Neural Rankers provide superior accuracy and scaling potential when handling massive datasets and complex embedding features, albeit at a higher specialized hardware cost.
Introduction
In large-scale recommendation pipelines, the "Ranking" stage serves as the final arbiter of user experience. After a retrieval layer filters millions of items down to a few thousand candidates, the ranking model must precisely predict user engagement to determine the final display order. For over a decade, Gradient Boosted Decision Trees (GBDTs) have dominated this domain due to their effectiveness on tabular data. However, the emergence of Deep Learning has introduced Neural Rankers as a powerful alternative.
This study moves beyond academic benchmarks to provide a pragmatic industry perspective from ShareChat, one of India's largest social media platforms. We examine the trade-offs between the reliability of GBDTs and the flexibility of Neural Networks, focusing on accuracy, training throughput, and engineering complexity in a production environment.
Methodology
Model Architectures
We compared two primary modeling paradigms:
- Neural Ranker (MMoE): We implemented a Multi-gate Mixture of Experts (MMoE) architecture. This Multi-Task Learning (MTL) framework is specifically designed to predict multiple, often conflicting, user signals (e.g., likes, shares, and favorites) simultaneously by sharing sub-networks (experts) while maintaining task-specific gates.
- GBDT (Catboost): We utilized Catboost as our GBDT representative. Catboost was selected for its native handling of categorical variables and its relatively robust performance with embedding features, which are typically challenging for tree-based models.
Objective Function
To ensure a fair comparison, both models were optimized using a pointwise multi-objective logloss (cross-entropy) function. The models were tasked with predicting the likelihood of four key engagement signals: Like, Share, Favorite, and Video Play.
Experimental Setup
Dataset Scale
The study utilized ShareChat’s internal production data, which significantly exceeds the scale of standard public benchmarks:
- Volume: Approximately 2 billion training data points collected over a 7-day window.
- Features: Roughly 500 features, including user/item embeddings, historical engagement metrics, and demographic data.
- Constraints: For GBDT training, the data required downsampling to maintain manageable training times, highlighting a significant practical limitation of the GBDT paradigm at the billion-row scale.
Comparative Analysis and Insights
The Scaling Advantage
Neural Networks act as universal function approximators that thrive on data volume. Our experiments revealed that while GBDTs and Neural Rankers perform similarly on smaller datasets (around 20 million points), their trajectories diverge as the data scales. The Neural Ranker’s performance continued to improve as we scaled to billions of points, whereas the GBDT performance began to plateau.
Feature Consumption and Embeddings
A critical finding involved the handling of historical features. Neural Rankers demonstrated a superior ability to "consume" complex, learnable embedding features and user history. GBDTs struggled to perform the complex aggregations required to extract value from these high-dimensional representations, which Neural Networks handle natively through backpropagation.
Hardware and Engineering Velocity
The performance gains of Neural Rankers come with a "Hardware Tax." These models require specialized infrastructure, such as TPUs or high-end GPUs. In contrast, GBDTs are less reliant on specialized hardware, offering a potentially higher "accuracy-per-dollar" ratio for smaller organizations.
From an engineering perspective, neural frameworks like TensorFlow offer streamlined distribution strategies for TPUs. This simplifies the scaling of the training process compared to GBDTs, which often require complex integrations with big-data processing tools like Apache Beam to handle datasets of this magnitude.
Key Results
Our evaluation across all metrics (Logloss, ROC-AUC, and PR-AUC) showed that the MMoE model consistently outperformed Catboost.
- Accuracy: On the "Like" signal, the MMoE achieved a ROC-AUC of 0.9420, surpassing Catboost’s 0.9388.
- Training Efficiency: Utilizing a v2-32 TPU allowed the MMoE to train nearly twice as fast as a v2-8 configuration. While this increased the total cost, it significantly reduced the iteration cycle for researchers.
- Data Saturation: At the 9-billion-point scale (tested specifically for MMoE), the neural architecture showed marginal improvements that were unattainable for the GBDT due to engineering constraints.
Conclusion and Future Work
This case study demonstrates that for massive-scale recommendation tasks, Neural Rankers provide a clear advantage in accuracy and scalability. However, GBDTs remain a highly effective and cost-efficient baseline for smaller datasets or organizations with limited access to specialized compute resources.
While this study focused on offline metrics, future research will explore the transition to Online Learning. Unlike GBDTs, which are difficult to update incrementally, Neural Networks are naturally suited for continuous training. We aim to evaluate whether the "Neural Paradigm" maintains its lead in a live production environment where models must adapt to real-time user behavior.
Summary updated