λOpt: Learn to Regularize Recommender Models in Finer Levels

Yihong Chen, Bei Chen, Xiangnan He, Chen Gao, Yong Li, Jian-Guang Lou, Yue Wang

Abstract

Recommendation models mainly deal with categorical variables, such as user/item ID and attributes. Besides the high-cardinality issue, the interactions among such categorical variables are usually long-tailed, with the head made up of highly frequent values and a long tail of rare ones. This phenomenon results in the data sparsity issue, making it essential to regularize the models to ensure generalization. The common practice is to employ grid search to manually tune regularization hyperparameters based on the validation data. However, it requires non-trivial efforts and large computation resources to search the whole candidate space; even so, it may not lead to the optimal choice, for which different parameters should have different regularization strengths. In this paper, we propose a hyperparameter optimization method, lambdaOpt, which automatically and adaptively enforces regularization during training. Specifically, it updates the regularization coefficients based on the performance of validation data. With lambdaOpt, the notorious tuning of regularization hyperparameters can be avoided; more importantly, it allows fine-grained regularization (i.e. each parameter can have an individualized regularization coefficient), leading to better generalized models. We show how to employ lambdaOpt on matrix factorization, a classical model that is representative of a large family of recommender models. Extensive experiments on two public benchmarks demonstrate the superiority of our method in boosting the performance of top-K recommendation.

The Regularization Paradox in Recommender Systems

Recommender systems rely heavily on categorical data, such as User and Item IDs. This data typically follows a "long-tailed" distribution: a small number of "head" items receive the vast majority of interactions, while a "long tail" of items has very few.

This distribution creates a significant challenge for regularization. A high regularization penalty ($\lambda$) prevents overfitting on rare items but causes "underfitting" on popular ones, losing valuable nuances. Conversely, a low $\lambda$ models popular items well but overfits to noise in sparse data. Traditionally, practitioners use "grid search"—running numerous full training cycles with different fixed $\lambda$ values—to find a compromise. This process is computationally expensive and suboptimal because it applies a uniform penalty across all parameters.

The λOpt Framework

The proposed λOpt framework treats the regularization coefficient not as a fixed hyperparameter, but as a parameter to be learned via bi-level optimization.

Comparison of fixed vs. adaptive regularization approaches

As shown in the comparison above, the traditional fixed approach (a) requires searching through a discrete space of $\lambda$ values, each requiring a separate optimization of model parameters $\theta$. In contrast, the adaptive approach (b) updates $\lambda$ and $\theta$ concurrently.

  1. The Inner Loop: The model updates its standard parameters (e.g., user/item embeddings) to minimize training loss using the current $\lambda$.
  2. The Outer Loop: The system calculates the gradient of the validation loss with respect to $\lambda$ by looking "one step ahead" at how a gradient step would affect performance. It then updates $\lambda$ accordingly.

Fine-Grained Regularization

A key advantage of λOpt is its ability to handle fine-grained regularization. While previous methods might tune a single $\lambda$ or one per dimension, λOpt can optimize User-wise, Item-wise, and Dimension-wise regularization simultaneously.

Computational graph of λOpt for Matrix Factorization

The computational graph illustrates how the regularization term is decomposed into specific coefficients ($\Lambda^I_{i,:}$, $\Lambda^U_{u,:}$, $\Lambda^I_{j,:}$) for positive items, users, and negative items. By leveraging Automatic Differentiation, λOpt remains compatible with modern optimizers like Adam, overcoming the limitations of previous adaptive methods restricted to simple SGD.

Experimental Evaluation

The researchers evaluated λOpt on two benchmarks: Amazon Food Review (highly sparse, ~39k interactions) and MovieLens 10M (large-scale, ~10M interactions).

Key Insights

  • Personalized Penalties: λOpt automatically assigns higher penalties to active users (who risk overfitting) and lower penalties to sparse users, allowing the model to learn more effectively from limited data.
  • Dynamic Trajectory: Regularization is not static. λOpt discovers a trajectory where penalties are low initially to capture basic signals and increase as the model converges to prevent memorizing noise.
  • Efficiency: By tuning $\lambda$ during a single training run, λOpt avoids the massive overhead of grid search, typically costing only 3x the computation of a standard training run.

Performance Results

On the Amazon Food dataset, the fine-grained version of λOpt (DUI: Dimension-User-Item) outperformed the best possible fixed-$\lambda$ model by 26% in Hit Ratio (HR@100) and 31% in NDCG@100. It consistently surpassed state-of-the-art methods like Adversarial Matrix Factorization (AMF) and Neural Collaborative Filtering (NeuMF), particularly benefiting users with few interactions.

Discussion and Future Work

While λOpt demonstrates significant gains, it relies on a representative validation set to guide the learning of $\lambda$. Future work includes extending the framework to more complex architectures like Factorization Machines (FM) and establishing formal theoretical convergence properties for bi-level optimization in non-convex settings.

Summary updated

Abstract

Recommendation models mainly deal with categorical variables, such as user/item ID and attributes. Besides the high-cardinality issue, the interactions among such categorical variables are usually long-tailed, with the head made up of highly frequent values and a long tail of rare ones. This phenomenon results in the data sparsity issue, making it essential to regularize the models to ensure generalization. The common practice is to employ grid search to manually tune regularization hyperparameters based on the validation data. However, it requires non-trivial efforts and large computation resources to search the whole candidate space; even so, it may not lead to the optimal choice, for which different parameters should have different regularization strengths. In this paper, we propose a hyperparameter optimization method, lambdaOpt, which automatically and adaptively enforces regularization during training. Specifically, it updates the regularization coefficients based on the performance of validation data. With lambdaOpt, the notorious tuning of regularization hyperparameters can be avoided; more importantly, it allows fine-grained regularization (i.e. each parameter can have an individualized regularization coefficient), leading to better generalized models. We show how to employ lambdaOpt on matrix factorization, a classical model that is representative of a large family of recommender models. Extensive experiments on two public benchmarks demonstrate the superiority of our method in boosting the performance of top-K recommendation.
BibTeX
@inproceedings{chen2019opt,
  author = {Yihong Chen and Bei Chen and Xiangnan He and Chen Gao and Yong Li and Jian-Guang Lou and Yue Wang},
  title = {{λOpt: Learn to Regularize Recommender Models in Finer Levels}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2019},
  url = {https://openreview.net/forum?id=_Wp1RueEbr},
}
Figure