A Boosting Framework of Factorization Machine

Longfei Li, Peilin Zhao, Jun Zhou, Xiaolong LiarXiv 2018 · OpenReview

Abstract

Recently, Factorization Machines (FM) has become more and more popular for recommendation systems, due to its effectiveness in finding informative interactions between features. Usually, the weights for the interactions is learnt as a low rank weight matrix, which is formulated as an inner product of two low rank matrices. This low rank can help improve the generalization ability of Factorization Machines. However, to choose the rank properly, it usually needs to run the algorithm for many times using different ranks, which clearly is inefficient for some large-scale datasets. To alleviate this issue, we propose an Adaptive Boosting framework of Factorization Machines (AdaFM), which can adaptively search for proper ranks for different datasets without re-training. Instead of using a fixed rank for FM, the proposed algorithm will adaptively gradually increases its rank according to its performance until the performance does not grow, using boosting strategy. To verify the performance of our proposed framework, we conduct an extensive set of experiments on many real-world datasets. Encouraging empirical results shows that the proposed algorithms are generally more effective than state-of-the-art other Factorization Machines.

Introduction

Factorization Machines (FMs) are a staple in recommendation systems because they excel at capturing feature interactions in highly sparse data. However, they face a significant challenge in rank selection: the latent factor dimension (the rank $k$) must be chosen carefully. If $k$ is too small, the model underfits; if it's too large, it overfits and becomes computationally expensive. Currently, the optimal rank is typically found through trial and error—running the algorithm multiple times with different ranks and re-training from scratch each time. This process is inefficient for large-scale datasets. Furthermore, standard FMs optimized for rating prediction do not always translate to superior ranking performance in top-N recommendation tasks.

The AdaFM Framework

The proposed AdaFM (Adaptive Boosting Factorization Machine) treats rank selection as an additive boosting process. Instead of training a single high-rank FM, AdaFM constructs an ensemble in rounds:

  1. Component Training: In each round, a "weak" FM with a small, fixed rank (e.g., $k=2$ or $3$) is trained.
  2. Adaptive Weighting: Similar to AdaBoost, the framework maintains weights for the training data, assigning higher importance to "hard" users whose item lists are currently poorly ranked.
  3. Ensemble Construction: The final model is a linear combination of these component FMs.

A key mathematical insight is that a linear combination of $T$ Factorization Machines, each with rank $k$, is mathematically equivalent to a single Factorization Machine with rank $T \times k$. This allows the model to "grow" its rank adaptively until performance saturates, eliminating the need for manual grid search.

The framework is flexible and supports various base learners:

  • AdaFM-O: Original FM for rating prediction.
  • AdaFM-P: Pairwise Ranking FM optimizing for AUC.
  • AdaFM-S/D/W: Variants of LambdaFM optimizing for rank-biased metrics like NDCG.

Experimental Evaluation

The authors evaluated AdaFM on three real-world datasets: Yelp, Lastfm, and Yahoo Music. The experiments demonstrate that AdaFM consistently outperforms base FMs and other state-of-the-art variants.

Key Results

  • Superior Ranking Performance: AdaFM-P achieved significant improvements in AUC over standard Pairwise Ranking FM. For instance, on the Lastfm dataset, the boosting approach shows a clear performance gap over the baseline FM as the effective rank increases.
AUC performance on the Lastfm dataset comparing standard FM with AdaFM variants.
  • Parameter Efficiency: An AdaFM ensemble with a total effective rank of 8 often outperformed a standard FM with $k=20$, suggesting that the boosting approach finds a more effective representation than standard SGD training on a high-rank matrix.
  • NDCG Gains: On datasets like Yelp, boosted LambdaFM variants (AdaFM-D) achieved the highest NDCG scores, proving effective at optimizing the top of recommendation lists.
NDCG performance of AdaFM-D variants compared to LFM-D as the effective rank K increases.

Conclusion

AdaFM simplifies the rank selection problem by adaptively searching the rank space through boosting. By focusing on "hard" users and incrementally increasing model complexity, it achieves state-of-the-art performance with greater efficiency than traditional grid search methods. Future work includes extending this framework to higher-order FMs and DeepFM architectures.

Summary updated

Abstract

Recently, Factorization Machines (FM) has become more and more popular for recommendation systems, due to its effectiveness in finding informative interactions between features. Usually, the weights for the interactions is learnt as a low rank weight matrix, which is formulated as an inner product of two low rank matrices. This low rank can help improve the generalization ability of Factorization Machines. However, to choose the rank properly, it usually needs to run the algorithm for many times using different ranks, which clearly is inefficient for some large-scale datasets. To alleviate this issue, we propose an Adaptive Boosting framework of Factorization Machines (AdaFM), which can adaptively search for proper ranks for different datasets without re-training. Instead of using a fixed rank for FM, the proposed algorithm will adaptively gradually increases its rank according to its performance until the performance does not grow, using boosting strategy. To verify the performance of our proposed framework, we conduct an extensive set of experiments on many real-world datasets. Encouraging empirical results shows that the proposed algorithms are generally more effective than state-of-the-art other Factorization Machines.
BibTeX
@misc{li2018boosting,
  author = {Longfei Li and Peilin Zhao and Jun Zhou and Xiaolong Li},
  title = {{A Boosting Framework of Factorization Machine}},
  year = {2018},
  eprint = {1804.06027},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.06027},
}

Loading related content...

Figure