Enhancing Factorization Machines with Generalized Metric Learning

Yangyang Guo, Zhiyong Cheng, Jiazheng Jing, Yanpeng Lin, Liqiang Nie, Meng Wang

Abstract

Factorization Machines (FMs) are effective in incorporating side information to overcome the cold-start and data sparsity problems in recommender systems. Traditional FMs adopt the inner product to model the second-order interactions between different attributes, which are represented via feature vectors. The problem is that the inner product violates the triangle inequality property of feature vectors. As a result, it cannot well capture fine-grained attribute interactions, resulting in sub-optimal performance. Recently, the Euclidean distance is exploited in FMs to replace the inner product and has delivered better performance. However, previous FM methods including the ones equipped with the Euclidean distance all focus on the attribute-level interaction modeling, ignoring the critical intrinsic feature correlations inside attributes. Thereby, they fail to model the complex and rich interactions exhibited in the real-world data. To tackle this problem, in this paper, we propose a FM framework equipped with generalized metric learning techniques to better capture these feature correlations. In particular, based on this framework, we present a Mahalanobis distance and a deep neural network (DNN) methods, which can effectively model the linear and non-linear correlations between features, respectively. Besides, we design an efficient approach for simplifying the model functions. Experiments on several benchmark datasets demonstrate that our proposed framework outperforms several state-of-the-art baselines by a large margin. Moreover, we collect a new large-scale dataset on second-hand trading to justify the effectiveness of our method over cold-start and data sparsity problems in recommender systems.

Introduction

Recommender systems frequently encounter challenges such as "cold-start" for new users or items and extreme data sparsity. Factorization Machines (FMs) have become a staple solution by utilizing side information (e.g., user demographics or item categories) to predict interactions. However, traditional FMs suffer from two primary limitations. First, the reliance on the inner product to model attribute interactions violates the triangle inequality, which is essential for representing fine-grained similarities in latent spaces. Second, existing models typically focus on inter-attribute interactions (e.g., User ID vs. Item ID) while neglecting intra-attribute correlations—the intrinsic relationships between features within a single attribute.

Linear vs. Non-linear correlations in feature interactions.

As shown in the figure above, real-world data often exhibits complex linear and non-linear correlations that simple inner-product-based models fail to capture accurately. By ignoring these complex, non-linear "feature-level" interactions, traditional models leave significant performance on the table.

The GML-FM Framework

To address these shortcomings, we introduce the Generalized Metric Learning Factorization Machine (GML-FM). This framework replaces the standard inner product with a generalized distance function $D(v_i, v_j)$, allowing for more robust similarity modeling.

Core Innovations

  • Transformation Weights ($w_{ij}$): Since distance functions are non-negative, we introduce learnable weights for each interaction. This enables the model to map interactions across the entire real-number space, maintaining the flexibility of traditional FMs.
  • Mahalanobis Distance: This variant incorporates a positive semi-definite matrix to project features into a new space, effectively capturing linear correlations between features.
  • DNN-based Distance: To model highly complex, non-linear correlations, we employ a Deep Neural Network (MLP) to transform feature vectors before distance calculation.
Standard Factorization Machine (FM) architecture using inner product interactions.

While traditional FMs (illustrated above) rely on simple inner products, GML-FM generalizes this interaction to capture deeper feature-level relationships. Furthermore, we optimize the computational complexity from $O((kn)^2)$ to $O(k^2n)$, ensuring that GML-FM remains as efficient as its predecessors.

Experimental Evaluation

We evaluated GML-FM against state-of-the-art baselines (e.g., DeepFM, NFM, TransFM) using several benchmarks, including Amazon Product Data and MovieLens 1M. We also introduced the Mercari Dataset, a massive second-hand trading dataset with 265.4 million items, specifically designed to test performance under extreme sparsity.

Key Results

  • Superior Performance: The DNN-based GML-FM consistently outperformed all baselines. Notably, on the sparse Mercari-Ticket dataset, it achieved a 16.13% absolute improvement in Hit Ratio (HR@10).
  • Robustness to Cold-Start: GML-FM demonstrated significant advantages over specialized meta-learning models like MAMO as the number of available interactions decreased.
  • Latent Space Clustering: T-SNE visualizations confirmed that GML-FM clusters positive user-item pairs more tightly than inner-product models, which often produce noisy, scattered distributions.

Conclusion

GML-FM advances the state of recommendation systems by integrating generalized metric learning into the FM framework. By accounting for both linear and non-linear intra-attribute correlations and satisfying the triangle inequality, GML-FM provides a more accurate and robust representation of feature interactions. Future work will explore ranking-based loss functions and the application of this framework to large-scale product search.

Summary updated

Abstract

Factorization Machines (FMs) are effective in incorporating side information to overcome the cold-start and data sparsity problems in recommender systems. Traditional FMs adopt the inner product to model the second-order interactions between different attributes, which are represented via feature vectors. The problem is that the inner product violates the triangle inequality property of feature vectors. As a result, it cannot well capture fine-grained attribute interactions, resulting in sub-optimal performance. Recently, the Euclidean distance is exploited in FMs to replace the inner product and has delivered better performance. However, previous FM methods including the ones equipped with the Euclidean distance all focus on the attribute-level interaction modeling, ignoring the critical intrinsic feature correlations inside attributes. Thereby, they fail to model the complex and rich interactions exhibited in the real-world data. To tackle this problem, in this paper, we propose a FM framework equipped with generalized metric learning techniques to better capture these feature correlations. In particular, based on this framework, we present a Mahalanobis distance and a deep neural network (DNN) methods, which can effectively model the linear and non-linear correlations between features, respectively. Besides, we design an efficient approach for simplifying the model functions. Experiments on several benchmark datasets demonstrate that our proposed framework outperforms several state-of-the-art baselines by a large margin. Moreover, we collect a new large-scale dataset on second-hand trading to justify the effectiveness of our method over cold-start and data sparsity problems in recommender systems.
BibTeX
@article{guo2022enhancing,
  author = {Yangyang Guo and Zhiyong Cheng and Jiazheng Jing and Yanpeng Lin and Liqiang Nie and Meng Wang},
  title = {{Enhancing Factorization Machines with Generalized Metric Learning}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2022},
  doi = {10.1109/tkde.2020.3034613},
}
Figure