InteractionNN: A Neural Network for Learning Hidden Features in Sparse Prediction

Xiaowang Zhang, Qiang Gao, Zhiyong FengIJCAI 2019 · OpenReview

Abstract

In this paper, we present a neural network (InteractionNN) for sparse predictive analysis where hidden features of sparse data can be learned by multilevel feature interaction. To characterize multilevel interaction of features, InteractionNN consists of three modules, namely, nonlinear interaction pooling, layer-lossing, and embedding. Nonlinear interaction pooling (NI pooling) is a hierarchical structure and, by shortcut connection, constructs low-level feature interactions from basic dense features to elementary features. Layer-lossing is a feed-forward neural network where high-level feature interactions can be learned from low-level feature interactions via correlation of all layers with target. Moreover, embedding is to extract basic dense features from sparse features of data which can help in reducing our proposed model computational complex. Finally, our experiment evaluates on the two benchmark datasets and the experimental results show that InteractionNN performs better than most of state-of-the-art models in sparse regression.

Introduction

In recommendation systems and click-through rate (CTR) prediction, data is notoriously sparse. One-hot encoding categorical variables like user ID or location results in massive vectors dominated by zeros. Extracting meaningful "hidden features" from this sparse data is critical for predictive accuracy. InteractionNN, presented at IJCAI-19, addresses this by focusing on feature transitivity and reusability through a novel architecture.

Existing models like Factorization Machines (FM) excel at low-level interactions but struggle with complex high-level patterns. Conversely, standard deep neural networks often lose the "history" of lower-level features as data progresses through layers. InteractionNN bridges this gap by ensuring that low-level interactions are preserved and that high-level layers remain relevant to the prediction target.

Methodology

InteractionNN is composed of three primary modules: Hybrid Embedding, Nonlinear Interaction (NI) Pooling, and Layer-lossing.

Hybrid Embedding

The model converts sparse input into dense representations, creating two types of embeddings: a dense vector for linear features and a dense matrix for nonlinear feature interactions. This dual approach helps manage computational complexity while preparing the data for multilevel interaction.

Nonlinear Interaction (NI) Pooling

The NI pooling module serves as the "low-level" engine of the network. Inspired by ResNet, it utilizes shortcut connections to pass features from earlier stages directly to later ones, preserving the transitivity of feature relationships.

The Nonlinear Interaction (NI) Pooling architecture featuring shortcut connections and Hadamard products.

As shown in Figure 6, the module employs two key operators:

  • Column Sum ($\oplus$): Captures statistical information across the embedding matrix.
  • Hadamard Product ($\odot$): Strengthens features based on matrix factorization principles. The shortcuts ensure that if feature A interacts with B, and B with C, the relationship between A and C is maintained and reusable throughout the network.

Layer-lossing

While standard networks use only the final layer for prediction, InteractionNN introduces "Layer-lossing." In this mechanism, every hidden layer is connected to a linear model that attempts to predict the target.

The Layer-lossing mechanism where intermediate layers contribute to the final prediction loss.

As illustrated in Figure 5, the total loss function is the sum of the final prediction loss and the losses from each intermediate layer. This forces every layer to extract features relevant to the final goal, preventing the network from losing focus in deeper abstractions. Each layer also incorporates a "BAD" unit (Batch normalization, Activation, and Dropout) to improve convergence and prevent overfitting.

Experimental Evaluation

The model was evaluated on two benchmark datasets: Frappe (app usage) and MovieLens (movie ratings). InteractionNN consistently outperformed state-of-the-art models like FM and Neural Factorization Machines (NFM) across RMSE and AUC metrics.

Specifically, InteractionNN outperformed FM by 6.3% and NFM by 0.8%. The NI pooling module alone demonstrated a 4.4% improvement over the BI-Interaction pooling used in NFM, validating the shortcut-based hierarchical structure. The researchers also noted that while depth is useful, adding too many layers in extremely sparse settings can lead to diminishing returns due to gradient issues.

Conclusion

InteractionNN demonstrates that shortcut connections and intermediate supervision (Layer-lossing) are vital for sparse predictive analysis. By maintaining the "memory" of basic feature interactions and ensuring every layer contributes to the target prediction, the model effectively captures complex hidden features. Future work will focus on further automating the discovery of important feature interactions to reduce the need for manual feature engineering.

Summary updated

Abstract

In this paper, we present a neural network (InteractionNN) for sparse predictive analysis where hidden features of sparse data can be learned by multilevel feature interaction. To characterize multilevel interaction of features, InteractionNN consists of three modules, namely, nonlinear interaction pooling, layer-lossing, and embedding. Nonlinear interaction pooling (NI pooling) is a hierarchical structure and, by shortcut connection, constructs low-level feature interactions from basic dense features to elementary features. Layer-lossing is a feed-forward neural network where high-level feature interactions can be learned from low-level feature interactions via correlation of all layers with target. Moreover, embedding is to extract basic dense features from sparse features of data which can help in reducing our proposed model computational complex. Finally, our experiment evaluates on the two benchmark datasets and the experimental results show that InteractionNN performs better than most of state-of-the-art models in sparse regression.
BibTeX
@inproceedings{zhang2019interactionnn,
  author = {Xiaowang Zhang and Qiang Gao and Zhiyong Feng},
  title = {{InteractionNN: A Neural Network for Learning Hidden Features in Sparse Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=HA3zhP_k85},
}
Figure