Enhancing Factorization Machines With Generalized Metric Learning

Yangyang Guo, Zhiyong Cheng, Jiazheng Jing, Yanpeng Lin, Liqiang Nie, Meng Wang

TL;DR

Factorization Machines (FMs) are a staple in recommendation systems, but they often rely on inner products that fail to capture complex, fine-grained feature relationships. This paper introduces GML-FM, a framework that replaces the inner product with generalized metric learning (like Mahalanobis and deep neural distances) to better model both linear and non-linear correlations within and between attributes.

What Problem Does This Paper Address?

Recommender systems often struggle with "cold-start" (new users/items) and data sparsity. Factorization Machines solve this by using "side information" (like user age or item category) to predict interactions. However, traditional FMs have two major flaws:

  1. The Inner Product Limitation: They use the inner product to model how attributes interact. Mathematically, the inner product violates the triangle inequality, which is a property crucial for modeling fine-grained similarities in a vector space.
  2. Ignoring Intra-Attribute Correlations: Existing FMs focus on how different attributes interact (e.g., User ID vs. Item ID) but ignore the intrinsic correlations inside an attribute’s features (e.g., a movie's "acting quality" vs. "actor popularity").

By ignoring these complex, non-linear "feature-level" interactions, traditional models leave significant performance on the table.

Methods

The authors propose the Generalized Metric Learning Factorization Machine (GML-FM). The core innovation is replacing the standard inner product with a generalized distance function $D(v_i, v_j)$.

Key components include:

  • Transformation Weights ($w_{ij}$): Because distances are always non-negative, simply replacing the inner product with a distance function limits the model's range. The authors introduce a learnable weight for each interaction, allowing the model to represent the full real-number space.
  • Mahalanobis Distance: This variant uses a positive semi-definite matrix to project features into a new space, effectively capturing linear correlations between features.
  • DNN-based Distance: For non-linear correlations, they use a Deep Neural Network (MLP) to transform the feature vectors before calculating the distance.
  • Efficiency Optimization: Calculating these interactions can be computationally expensive—$O((kn)^2)$. The authors derived a mathematical simplification that reduces the complexity to $O(k^2n)$, making the model as efficient as standard FMs.

Datasets

The researchers evaluated GML-FM on several well-known benchmarks and one massive new dataset:

  • Amazon Product Data: Subsets for Auto, Office, and Clothing.
  • MovieLens 1M: A standard benchmark for movie recommendations.
  • Mercari Dataset: A newly introduced large-scale dataset from a second-hand trading platform. It contains 265.4 million items and rich side information (item condition, shipping duration, etc.). It is significantly sparser than MovieLens, making it a perfect "stress test" for recommendation algorithms.

Key Insights and Ideas

  • The "Aha" of Intra-Attribute Modeling: Most researchers treat an attribute as a single point. This paper argues that attributes are composed of features with internal relationships. Capturing these "intra-attribute" interactions is the key to better performance in sparse environments.
  • Metric Learning > Inner Product: By satisfying the triangle inequality, metric learning provides a more robust way to represent similarity than the inner product, especially when trying to group "similar" items together in a latent space.
  • Generalization Power: The authors theoretically prove that GML-FM is a "super-set" framework; by changing the distance function or weights, it can simplify back into a vanilla FM or the more recent TransFM.

Key Results

  • SOTA Performance: GML-FM (specifically the DNN version) consistently outperformed state-of-the-art baselines like DeepFM, NFM, and TransFM across all datasets.
  • Winning in the "Sparse" Category: On the highly sparse Mercari-Ticket dataset, GML-FM achieved a 16.13% absolute improvement in Hit Ratio (HR@10) over the best-performing baseline.
  • Cold-Start Superiority: In tests against MAMO (a specialized meta-learning model for cold starts), GML-FM showed a wider performance gap as the number of interactions decreased, proving it is exceptionally robust when data is scarce.
  • Visual Proof: T-SNE visualizations showed that GML-FM clusters positive items (items a user actually liked) much more tightly than inner-product-based models, which produced scattered, noisy patterns.

Limitations and Future Work

While GML-FM is powerful, the authors note a few areas for growth:

  • Overfitting in Deep Layers: The study found that while adding DNN layers helps, going beyond two layers often leads to overfitting due to the increased number of parameters.
  • Ranking Strategies: The current work uses a regression-based loss. The authors suggest integrating Bayesian Personalized Ranking (BPR)—a pair-wise learning approach—to further boost top-N recommendation accuracy.
  • Search Applications: They plan to adapt the GML-FM method for product search tasks, where user-submitted queries provide another layer of complex feature interactions to model.

Summary updated

Abstract

Factorization Machines (FMs) are effective in incorporating side information to overcome the cold-start and data sparsity problems in recommender systems. Traditional FMs adopt the inner product to model the second-order interactions between different attributes, which are represented via feature vectors. The problem is that the inner product violates the triangle inequality property of feature vectors. As a result, it cannot well capture fine-grained attribute interactions, resulting in sub-optimal performance. Recently, the euclidean distance is exploited in FMs to replace the inner product and has delivered better performance. However, previous FM methods including the ones equipped with the euclidean distance all focus on the attribute-level interaction modeling, ignoring the critical intrinsic feature correlations inside attributes. Thereby, they fail to model the complex and rich interactions exhibited in the real-world data. To tackle this problem, in this paper, we propose a FM framework equipped with generalized metric learning techniques to better capture these feature correlations. In particular, based on this framework, we present a Mahalanobis distance and a deep neural network (DNN) methods, which can effectively model the linear and non-linear correlations between features, respectively. Besides, we design an efficient approach for simplifying the model functions. Experiments on several benchmark datasets demonstrate that our proposed framework outperforms several state-of-the-art baselines by a large margin. Moreover, we collect a new large-scale dataset on second-hand trading to justify the effectiveness of our method over cold-start and data sparsity problems in recommender systems.
BibTeX
@article{guo2022enhancing,
  author = {Yangyang Guo and Zhiyong Cheng and Jiazheng Jing and Yanpeng Lin and Liqiang Nie and Meng Wang},
  title = {{Enhancing Factorization Machines With Generalized Metric Learning}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2022},
  doi = {10.1109/tkde.2020.3034613},
}

Loading related content...

Figure