LambdaOpt: Learn to Regularize Recommender Models in Finer Levels

Yihong Chen, Bei Chen, Xiangnan He, Chen Gao, Yong Li, Jian-Guang Lou, Yue Wang

Abstract

Recommendation models mainly deal with categorical variables, such as user/item ID and attributes. Besides the high-cardinality issue, the interactions among such categorical variables are usually long-tailed, with the head made up of highly frequent values and a long tail of rare ones. This phenomenon results in the data sparsity issue, making it essential to regularize the models to ensure generalization. The common practice is to employ grid search to manually tune regularization hyperparameters based on the validation data. However, it requires non-trivial efforts and large computation resources to search the whole candidate space; even so, it may not lead to the optimal choice, for which different parameters should have different regularization strengths. In this paper, we propose a hyperparameter optimization method, LambdaOpt, which automatically and adaptively enforces regularization during training. Specifically, it updates the regularization coefficients based on the performance of validation data. With LambdaOpt, the notorious tuning of regularization hyperparameters can be avoided; more importantly, it allows fine-grained regularization (i.e. each parameter can have an individualized regularization coefficient), leading to better generalized models. We show how to employ LambdaOpt on matrix factorization, a classical model that is representative of a large family of recommender models. Extensive experiments on two public benchmarks demonstrate the superiority of our method in boosting the performance of top-K recommendation.

Introduction

In modern recommendation systems, models must handle massive categorical variables, such as User and Item IDs. These variables typically follow a "long-tailed" distribution: a small number of popular items account for the majority of interactions, while the vast majority of items are sparse, appearing in only a few records. This sparsity makes models prone to overfitting, where they memorize noise rather than learning generalizable patterns.

To mitigate this, researchers employ $L_2$ regularization, controlled by a hyperparameter $\lambda$. However, traditional regularization faces two significant hurdles. First, the "One Size Fits All" fallacy assumes a single global $\lambda$ is optimal for all users and items, ignoring that a frequent user requires different regularization than a rare one. Second, the "Tuning Nightmare" of grid search requires training models hundreds of times to find a suitable $\lambda$, which is computationally expensive and often yields suboptimal results.

Impact of different regularization strengths (λ) on model performance (AUC) over training epochs.

The LambdaOpt Framework

The authors introduce $\lambda$Opt, an automated framework that treats regularization coefficients as learnable parameters updated during training. This approach utilizes bi-level optimization to move beyond static hyperparameters.

Adaptive $\Lambda$-Trajectory

Unlike fixed approaches, $\lambda$Opt updates $\lambda$ at every training step. This creates an adaptive trajectory where regularization can start low to facilitate initial learning and increase as the model converges to prevent overfitting.

Fine-Grained Regularization

The core innovation of $\lambda$Opt is its ability to assign individualized regularization coefficients to specific users, items, or even specific dimensions of embedding vectors. This "fine-grained" approach allows the model to tailor its complexity to the specific data density of each entity.

Bi-Level Optimization Loop

The framework operates through two nested loops:

  1. Inner Loop: Updates model parameters (e.g., embeddings) on the training set using the current $\lambda$.
  2. Outer Loop: Updates the $\lambda$ values by calculating the gradient of the validation loss with respect to $\lambda$.
Comparison between the traditional fixed hyperparameter approach (a) and the proposed adaptive λOpt approach (b).

By leveraging automatic differentiation, $\lambda$Opt remains compatible with modern adaptive optimizers like Adam and Adagrad, overcoming limitations of previous adaptive regularization methods.

Experimental Results

The researchers evaluated $\lambda$Opt on two benchmarks: Amazon Food Review and MovieLens 10M.

Key Insights

  • Personalized Regularization: $\lambda$Opt learns to apply stronger regularization to active users (who are prone to overfitting) and weaker regularization to sparse users, who need the model to rely more on their limited data.
  • Efficiency: While a single $\lambda$Opt step is roughly 3x more expensive than a standard step, it only requires a single training run. This is significantly more efficient than a grid search requiring 100+ runs.
  • Performance Gains: On the Amazon Food dataset, the fine-grained variant (MF-$\lambda$Opt-DUI) achieved a Hit Ratio (HR@100) of 0.5251, a 25% relative improvement over the fixed-$\lambda$ baseline (0.4163).

Conclusion

$\lambda$Opt effectively automates the tuning of regularization hyperparameters while enabling a level of granularity impossible with manual search. By adaptively learning regularization strengths, it significantly improves the accuracy and generalization of recommendation models, particularly in the face of extreme data sparsity.

Summary updated

Abstract

Recommendation models mainly deal with categorical variables, such as user/item ID and attributes. Besides the high-cardinality issue, the interactions among such categorical variables are usually long-tailed, with the head made up of highly frequent values and a long tail of rare ones. This phenomenon results in the data sparsity issue, making it essential to regularize the models to ensure generalization. The common practice is to employ grid search to manually tune regularization hyperparameters based on the validation data. However, it requires non-trivial efforts and large computation resources to search the whole candidate space; even so, it may not lead to the optimal choice, for which different parameters should have different regularization strengths. In this paper, we propose a hyperparameter optimization method, LambdaOpt, which automatically and adaptively enforces regularization during training. Specifically, it updates the regularization coefficients based on the performance of validation data. With LambdaOpt, the notorious tuning of regularization hyperparameters can be avoided; more importantly, it allows fine-grained regularization (i.e. each parameter can have an individualized regularization coefficient), leading to better generalized models. We show how to employ LambdaOpt on matrix factorization, a classical model that is representative of a large family of recommender models. Extensive experiments on two public benchmarks demonstrate the superiority of our method in boosting the performance of top-K recommendation.
BibTeX
@inproceedings{chen2019lambdaopt,
  author = {Yihong Chen and Bei Chen and Xiangnan He and Chen Gao and Yong Li and Jian-Guang Lou and Yue Wang},
  title = {{LambdaOpt: Learn to Regularize Recommender Models in Finer Levels}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2019},
  url = {https://openreview.net/forum?id=_Wp1RueEbr},
}
Figure