Boosting Factorization Machines via Saliency-Guided Mixup

Chenwang Wu, Defu Lian, Yong Ge, Min Zhou, Enhong Chen, Dacheng TaoIEEE TPAMI 2022 · OpenReview

TL;DR

This paper introduces MixFM and SMFM, novel methods that augment training data for Factorization Machines (FMs) to improve their ability to learn interactions between non-interactive features. By generating synthetic data, these approaches enhance generalization and performance, with SMFM further refining this by using feature saliency to select more informative synthetic samples.

What Problem Does This Paper Address?

Recommender systems heavily rely on Factorization Machines (FMs) due to their efficiency and ability to handle sparse, categorical data. However, a key limitation of FMs is their struggle with "non-interactive" features – features that rarely appear together in the training data. FMs can only learn about these features indirectly through their embeddings, failing to capture their direct interactions. This deficiency limits their expressive power and can lead to suboptimal performance, especially when dealing with complex, real-world datasets where certain feature combinations are inherently rare (e.g., smoking in public places). Existing methods to address this often require costly manual data collection or domain expertise, which this paper aims to circumvent.

Methods

The paper proposes two main methods:

  • MixFM (Mixup-powered Factorization Machine): Inspired by the "Mixup" data augmentation technique from computer vision, MixFM generates new training samples by taking convex combinations of existing samples. This process is domain-knowledge-independent, meaning it doesn't require specialized information about feature types or positions. Crucially, if the original samples being mixed have non-interactive features, the resulting mixed sample can learn their direct interaction. MixFM optimizes for "vicinal risk minimization" (VRM), considering both original and augmented data.
  • SMFM (Saliency-guided Mixup Factorization Machine): Recognizing that randomly generated Mixup samples might sometimes be redundant or even detrimental, SMFM introduces a "saliency" mechanism. It first generates multiple candidate neighbors for a given sample and then uses a custom saliency function to select the most informative neighbor to add to the training set. This saliency is derived from how much a feature influences the model's loss, aiming to focus augmentation on features that are critical for decision-making but might be underrepresented.

The authors also provide the first theoretical generalization bound for FMs and prove that both MixFM and SMFM achieve a smaller generalization error bound compared to standard FMs.

Datasets

The proposed methods were evaluated on five real-world datasets:

  • Frappe: A context-aware mobile app dataset.
  • MovieLens: A movie rating dataset.
  • Pet Supplies (Pet), Electronics, Books: Datasets derived from Amazon customer reviews.

The paper does not introduce new benchmarks but uses established datasets commonly found in recommender system research.

Key Insights and Ideas

  • Data Augmentation for Feature Interactions: The core idea that augmenting sparse data with intelligently generated samples can directly improve the learning of feature interactions, especially for those rarely seen together, is a significant contribution.
  • Domain-Agnostic Augmentation: MixFM's ability to generate useful data without requiring expert knowledge or specific feature metadata is a practical advantage for real-world applications.
  • Saliency for Informative Augmentation: SMFM's use of feature saliency to guide data generation is a clever way to ensure that the augmented data is not just random noise but actively contributes to learning. This moves beyond simple convex combinations to a more targeted approach.
  • Theoretical Generalization Bounds: Providing the first generalization bound for FMs is a substantial theoretical contribution, offering insights into how data size and embedding dimensions impact model performance.

Key Results

  • Performance Improvement: Both MixFM and SMFM consistently outperformed the baseline FM and other FM variants across all five datasets in terms of AUC (Area Under the ROC Curve). SMFM generally achieved the best AUC scores.
  • LogLoss Performance: While SMFM excelled in AUC, MixFM sometimes showed better LogLoss performance, suggesting a trade-off that might depend on the specific task requirements (prediction accuracy vs. model loss).
  • Effectiveness on FM Variants: The proposed strategies were successfully applied to other FM-based models (NFM, AFM, DCN), demonstrating their general applicability.
  • Robustness: MixFM showed improved robustness against noise perturbations compared to standard FM.
  • Training Time: While MixFM and SMFM increase training time due to data augmentation, the increase is linear with the amount of augmented data, and the inference time remains unchanged.

Limitations and Future Work

  • LogLoss vs. AUC Trade-off: The paper notes that SMFM's LogLoss performance can sometimes be inferior to MixFM, suggesting a potential trade-off between optimizing for AUC and LogLoss.
  • Generalizability to Other Models: While the authors suggest their methods are theoretically applicable to any differentiable model, they primarily focus on FMs and FM variants. Future work could explore their application to a broader range of recommendation models.
  • Dataset Size and Embedding Size Interaction: The optimal embedding size was found to interact with dataset size, with smaller datasets benefiting from larger embedding sizes (likely to capture more empirical error) and larger datasets benefiting from smaller embedding sizes (to avoid overfitting and improve generalization). This nuanced relationship could be further explored.
  • Potential for "Poisoning": The paper highlights that even "poisoning" data (i.e., adding mixed data) can be beneficial, suggesting a broader implication for data augmentation strategies.

Summary updated

Abstract

Factorization machines (FMs) are widely used in recommender systems due to their adaptability and ability to learn from sparse data. However, for the ubiquitous non-interactive features in sparse data, existing FMs can only estimate the parameters corresponding to these features via the inner product of their embeddings. Undeniably, they cannot learn the direct interactions of these features, which limits the model's expressive power. To this end, we first present MixFM, inspired by Mixup, to generate auxiliary training data to boost FMs. Unlike existing augmentation strategies that require labor costs and expertise to collect additional information such as position and fields, these augmented data are only by the convex combination of the raw ones without any professional knowledge support. More importantly, if non-interactive features exist in parent samples to be mixed respectively, MixFM will establish their direct interactions. Second, considering that MixFM may generate redundant or even detrimental instances, we further put forward a novel Factorization Machine powered by Saliency-guided Mixup (denoted as SMFM). Guided by the customized saliency, SMFM can generate more informative neighbor data. Through theoretical analysis, we prove that the proposed methods minimize the upper bound of the generalization error, which positively enhances FMs. Finally, extensive experiments on seven datasets confirm that our approaches are superior to baselines. Notably, the results also show that “poisoning” mixed data benefits the FM variants.
BibTeX
@article{wu2022boosting,
  author = {Chenwang Wu and Defu Lian and Yong Ge and Min Zhou and Enhong Chen and Dacheng Tao},
  title = {{Boosting Factorization Machines via Saliency-Guided Mixup}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=Jxm4nk6jEM},
}
Figure