Abstract
We address prediction problems on tabular categorical data, where each instance is defined by multiple categorical attributes (fields), and their categorical values (features). Such problems frequently arise in practical applications, including click-through rate prediction and social sciences. We introduce and analyze tensorFM, a new model that efficiently captures high-order interactions between attributes via a low-rank tensor approximation representing the strength of these interactions. Our model generalizes field-weighted factorization machines. Empirically, tensorFM demonstrates competitive performance with state-of-the-art methods. Additionally, its low latency makes it well-suited for time-sensitive applications, such as online advertising.
1. Introduction
In machine learning for tabular data—common in online advertising and social science databases—predictive accuracy often depends on how different attributes interact. For example, the interaction between a user's age and the time of day might be more predictive of an ad click than either feature alone. This paper introduces tensorFM, a model designed to capture these complex relationships more efficiently than existing methods.
2. Problem Statement: High-Order Interactions
Predicting outcomes on categorical tabular data requires modeling interactions between fields such as 'City', 'Device Type', or 'Occupation'. Standard Factorization Machines (FMs) effectively model pairwise (second-order) interactions but treat all field interactions with equal weight. Field-weighted Factorization Machines (FwFM) introduced interaction strength matrices but face scalability issues, as their complexity grows quadratically ($O(n^2)$) with the number of fields. Furthermore, they typically do not account for interactions beyond pairs (e.g., three-way or four-way interactions).
3. Methodology
The core innovation of tensorFM is the application of Canonical Polyadic (CP) decomposition to interaction tensors. Instead of storing a dense tensor of all possible $d$-way interaction strengths, tensorFM represents these strengths as a sum of rank-1 tensors. This low-rank approach reduces the computational complexity from $O(n^d)$ to linear scaling ($O(n)$) with respect to the number of fields. Specifically, with CP rank $r$, the inference complexity is $O(nkrd^2)$, ensuring scalability as the number of fields increases. Unlike previous high-order models, tensorFM learns the importance of specific field combinations directly from the data.
4. Experimental Results
The authors evaluated tensorFM on several benchmarks:
- Click-Through Rate Prediction: Using the Avazu and Criteo datasets, tensorFM achieved competitive AUC and LogLoss values, often outperforming HOFM and FwFM.
- Social Science Applications: On the COMPAS recidivism dataset, the model demonstrated its utility beyond advertising.
- Synthetic Benchmarks: tensorFM successfully recovered 3-way and 4-way interactions in noisy synthetic data, significantly outperforming standard FMs.
A key advantage of tensorFM is its efficiency. In empirical tests with 100 fields, tensorFM was approximately 10x faster than FwFM (0.001ms vs 0.012ms per instance), making it ideal for real-time, high-throughput environments.
5. Conclusion
tensorFM offers a scalable and interpretable approach to modeling high-order feature interactions. While currently a "shallow" model, it could potentially be integrated into deep "two-stream" architectures (e.g., DeepFM) to combine explicit interaction modeling with deep learning. Future research may also explore more sophisticated methods for handling continuous numerical data.
Summary updated