TL;DR
This paper introduces Loss-Balanced Small Perturbation Regularization (LSPR), a novel technique to improve the performance of industrial-scale ad ranking systems by making models more robust to small input perturbations. LSPR achieves this by treating perturbed data points with a reduced weight in the loss calculation, leading to better generalization and faster convergence compared to existing methods like Self-Consistency Regularization (SCR), and has been successfully deployed in a billion-scale system.
What Problem Does This Paper Address?
Online advertising relies heavily on sophisticated machine learning models to rank ads, aiming to show the most relevant ones to users to maximize engagement and click-through rates. These ranking systems are often massive, dealing with vast amounts of data and complex user interactions. A key challenge in training these large models, especially with sparse labels, is ensuring they generalize well and are robust to minor variations in input data. Traditional methods like self-consistency regularization (SCR) try to address this by enforcing similar predictions for slightly modified inputs. However, SCR can be computationally expensive and might not always be the most effective. This paper tackles the problem of improving the generalization and robustness of industrial-scale ad ranking models in a more efficient and effective way.
Methods
The core contribution of this paper is the introduction of Loss-Balanced Small Perturbation Regularization (LSPR). Unlike SCR, which adds an auxiliary loss to penalize differences between original and perturbed data predictions, LSPR takes a different approach. It generates perturbed data points by adding small noise to the original data. These perturbed points are then included in the training data, but crucially, they are assigned a smaller weight in the overall loss calculation. The intuition is that these slightly altered data points should still be predictable, but their influence on the model's learning should be down-weighted to avoid disrupting the learning dynamics too much.
The paper also discusses two key data augmentation techniques used to create these perturbations:
- Dropout: Applied to sparse features, this involves creating augmented copies of existing sparse features by randomly dropping some out.
- Gaussian Noise Injection: Applied to dense features, this involves adding random noise sampled from a Gaussian distribution to the feature values.
The authors also detail their integration of these perturbation-based regularization techniques into different stages of an industrial ad ranking system: retrieval, early-stage ranking, and final-stage ranking.
Datasets
The paper does not explicitly name specific public datasets used for training or evaluation. Instead, it refers to "industrial-scale experiments" and "an in-house dataset" that is described as having a "substantial volume of features, comprising thousands of dense features and hundreds of sparse features." They also mention experiments conducted on a "billion-scale industrial ranking system."
Key Insights and Ideas
- LSPR's efficiency: By down-weighting perturbed data in the loss, LSPR is less disruptive to the learning process than methods that enforce strict consistency, leading to potentially faster convergence and better alignment with optimal weights.
- Scalability is key: The paper emphasizes the successful deployment of LSPR in a massive, billion-scale industrial ad ranking system, highlighting its practical applicability and scalability.
- Balancing is crucial: Numerical analysis shows that the balance between the amount of noise introduced (
w) and the loss weight for perturbations (λ) is critical for the success of both LSPR and SCR. Smaller values for these weights are generally recommended. - Perturbation-based regularization is effective for ads ranking: The study demonstrates that these techniques, previously explored in other domains, can yield significant improvements in the context of industrial ad ranking.
Key Results
- Offline Experiments:
- Self-Consistency Regularization (SCR) achieved relative Normalized Entropy (NE) gains of approximately 0.1% to 0.3% across different stages of the ranking system (Retrieval, Early Stage, Final Stage).
- LSPR, when tested on the Final Ranker Stage, showed comparable or slightly better performance than SCR, achieving relative NE gains of 0.11% to 0.13% in various setups.
- Online Experiments: LSPR demonstrated a 0.1% to 0.2% relative improvement in online top-line metrics, which is considered significant for a billion-scale system.
- Numerical Analysis: LSPR showed better alignment with the optimal weight and faster convergence in weight space compared to SCR, especially with smaller perturbation weights.
Limitations and Future Work
- Hyperparameter Tuning: While LSPR is described as having only three major hyperparameters (dense feature perturb, sparse feature dropout, and loss weight), the paper notes that exploring perturbation-dependent weights is a worthwhile follow-up.
- Theoretical Understanding: The authors aim to push for deeper theoretical understanding of their methods, particularly concerning their industrial scalability.
- Unlabeled Data: Future research will focus on exploring other variations of using unlabeled data tailored for large-scale recommendation systems.
Summary updated