tensorFM: Low-Rank Approximations of Cross-Order Feature Interactions

Alessio Mazzetto, Mohammad Mahdi Khalili, Laura Fee Nern, Michael Viderman, Alex Shtoff, Krzysztof Dembczyński

Abstract

We address prediction problems on tabular categorical data, where each instance is defined by multiple categorical attributes (fields), and their categorical values (features). Such problems frequently arise in practical applications, including click-through rate prediction and social sciences. We introduce and analyze tensorFM, a new model that efficiently captures high-order interactions between attributes via a low-rank tensor approximation representing the strength of these interactions. Our model generalizes field-weighted factorization machines. Empirically, tensorFM demonstrates competitive performance with state-of-the-art methods. Additionally, its low latency makes it well-suited for time-sensitive applications, such as online advertising.

1. Introduction

In machine learning for tabular data—common in online advertising and social science databases—predictive accuracy often depends on how different attributes interact. For example, the interaction between a user's age and the time of day might be more predictive of an ad click than either feature alone. This paper introduces tensorFM, a model designed to capture these complex relationships more efficiently than existing methods.

2. Problem Statement: High-Order Interactions

Predicting outcomes on categorical tabular data requires modeling interactions between fields such as 'City', 'Device Type', or 'Occupation'. Standard Factorization Machines (FMs) effectively model pairwise (second-order) interactions but treat all field interactions with equal weight. Field-weighted Factorization Machines (FwFM) introduced interaction strength matrices but face scalability issues, as their complexity grows quadratically ($O(n^2)$) with the number of fields. Furthermore, they typically do not account for interactions beyond pairs (e.g., three-way or four-way interactions).

Examples of tabular categorical data from the COMPAS (left) and CTR (right) datasets used to evaluate tensorFM.

3. Methodology

The core innovation of tensorFM is the application of Canonical Polyadic (CP) decomposition to interaction tensors. Instead of storing a dense tensor of all possible $d$-way interaction strengths, tensorFM represents these strengths as a sum of rank-1 tensors. This low-rank approach reduces the computational complexity from $O(n^d)$ to linear scaling ($O(n)$) with respect to the number of fields. Specifically, with CP rank $r$, the inference complexity is $O(nkrd^2)$, ensuring scalability as the number of fields increases. Unlike previous high-order models, tensorFM learns the importance of specific field combinations directly from the data.

4. Experimental Results

The authors evaluated tensorFM on several benchmarks:

  • Click-Through Rate Prediction: Using the Avazu and Criteo datasets, tensorFM achieved competitive AUC and LogLoss values, often outperforming HOFM and FwFM.
  • Social Science Applications: On the COMPAS recidivism dataset, the model demonstrated its utility beyond advertising.
  • Synthetic Benchmarks: tensorFM successfully recovered 3-way and 4-way interactions in noisy synthetic data, significantly outperforming standard FMs.

A key advantage of tensorFM is its efficiency. In empirical tests with 100 fields, tensorFM was approximately 10x faster than FwFM (0.001ms vs 0.012ms per instance), making it ideal for real-time, high-throughput environments.

5. Conclusion

tensorFM offers a scalable and interpretable approach to modeling high-order feature interactions. While currently a "shallow" model, it could potentially be integrated into deep "two-stream" architectures (e.g., DeepFM) to combine explicit interaction modeling with deep learning. Future research may also explore more sophisticated methods for handling continuous numerical data.

Summary updated

Abstract

We address prediction problems on tabular categorical data, where each instance is defined by multiple categorical attributes, each taking values from a finite set. These attributes are often referred to as fields, and their categorical values as features. Such problems frequently arise in practical applications, including click-through rate prediction and social sciences. We introduce and analyze {tensorFM}, a new model that efficiently captures high-order interactions between attributes via a low-rank tensor approximation representing the strength of these interactions. Our model generalizes field-weighted factorization machines. Empirically, tensorFM demonstrates competitive performance with state-of-the-art methods. Additionally, its low latency makes it well-suited for time-sensitive applications, such as online advertising.
BibTeX
@misc{mazzetto2026tensorfm,
  author = {Alessio Mazzetto and Mohammad Mahdi Khalili and Laura Fee Nern and Michael Viderman and Alex Shtoff and Krzysztof Dembczyński},
  title = {{tensorFM: Low-Rank Approximations of Cross-Order Feature Interactions}},
  year = {2026},
  eprint = {2602.15229},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.15229},
}
Figure