Learning Feature Interactions with Lorentzian Factorization Machine

Canran Xu, Ming WuAAAI 2020 · DOI

In the domain of recommendation systems and click-through rate (CTR) prediction, the primary challenge lies in accurately modeling feature interactions. Traditional approaches, such as Factorization Machines (FM), rely on Euclidean inner products to capture these relationships. However, recent research suggests that Euclidean space may be too restrictive to represent the complex, hierarchical structures inherent in sparse feature data. The Lorentzian Factorization Machine (LorentzFM), introduced at AAAI-20, proposes a shift from flat Euclidean geometry to the curved manifold of hyperbolic space to achieve more expressive and efficient feature modeling.

The Limitations of Euclidean Geometry

Standard CTR models often utilize deep neural networks (e.g., DeepFM, xDeepFM) to learn higher-order feature interactions. While effective, these models are computationally expensive, parameter-heavy, and slow to train. The authors of LorentzFM argue that the fundamental limitation lies in the geometry itself. Euclidean space enforces the triangle inequality, which can restrict the model's ability to represent fine-grained or "spurious" feature relationships. By contrast, hyperbolic space expands exponentially, providing a more capacious environment for embedding sparse features.

Methodology: The LorentzFM Architecture

LorentzFM maps features onto an $n$-dimensional hyperboloid, a smooth, bowl-shaped surface within a Lorentzian manifold. This approach leverages two key innovations: Lorentzian embeddings and a specialized pooling mechanism.

Architecture of the LorentzFM model featuring the Triangle Pooling Layer.

1. Lorentzian Embeddings

Instead of standard vectors, features are represented as points on a hyperboloid. This geometry is naturally suited for capturing hierarchical structures and complex dependencies that are often present in recommendation datasets.

2. Triangle Pooling Layer

A core contribution of this work is the Triangle Pooling Layer. In hyperbolic space using the Lorentz distance, the triangle inequality can be violated. The model utilizes this property by calculating an interaction score based on how much a pair of feature vectors obeys or violates this geometric rule. This score serves as a robust measure of interaction validity without requiring the heavy $L_2$ regularization typical of Euclidean models.

3. Riemannian Optimization

To maintain the integrity of the embeddings on the curved manifold, the model employs Riemannian Stochastic Gradient Descent (RSGD). This optimization technique ensures that every parameter update keeps the feature vectors on the hyperboloid surface.

Experimental Results

The model was evaluated against several state-of-the-art baselines, including DeepFM, xDeepFM, and DCN, across four real-world datasets: Steam, MovieLens (1M), KKBox, and Avazu.

NDCG performance comparison across different embedding sizes on the Steam dataset.

Key Findings:

  • Superior Accuracy: LorentzFM consistently outperformed deep learning baselines in terms of MRR, AUC, and NDCG, particularly on extremely sparse datasets.
  • Efficiency: The model achieved these results with significantly fewer parameters—reducing the parameter count by up to 82.9% compared to xDeepFM.
  • Training Speed: Due to its lighter architecture (eliminating the need for deep MLP layers), training times were reduced by up to 69.7%.

Conclusion

LorentzFM demonstrates that the choice of geometry is a critical factor in recommendation performance. By moving to hyperbolic space, the model captures complex feature interactions more effectively than deep Euclidean models while remaining significantly more efficient. This work opens new avenues for exploring non-Euclidean geometries in large-scale machine learning tasks.

Summary updated

BibTeX
@inproceedings{xu2020learning,
  author = {Canran Xu and Ming Wu},
  title = {{Learning Feature Interactions with Lorentzian Factorization Machine}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.6119},
}
Figure