Abstract
Click-Through Rate prediction (CTR) is a crucial task in recommender systems, and it gained considerable attention in the past few years. The primary purpose of recent research emphasizes obtaining meaningful and powerful representations through mining low and high feature interactions using various components such as Deep Neural Networks (DNN), CrossNets, or transformer blocks. In this work, we propose the Deep Multi-Representation model (DeepMR) that jointly trains a mixture of two powerful feature representation learning components, namely DNNs and multi-head self-attentions. Furthermore, DeepMR integrates the novel residual with zero initialization (ReZero) connections to the DNN and the multi-head self-attention components for learning superior input representations. Experiments on three real-world datasets show that the proposed model significantly outperforms all state-of-the-art models in the task of click-through rate prediction.
Introduction
Predicting user engagement, specifically Click-Through Rate (CTR), is a cornerstone of modern digital economies, powering recommendation engines and online advertising platforms. The fundamental challenge in CTR prediction is modeling "feature interactions"—the non-linear relationships between disparate data fields such as user demographics, item categories, and temporal contexts.
Current research typically bifurcates into two methodologies: DNN-based models and Attention-based models. DNNs are proficient at capturing high-order, additive non-linear interactions but often struggle with multiplicative relationships. Conversely, attention mechanisms excel at capturing similarities and multiplicative interactions between fields but can be difficult to train as they scale in depth. Furthermore, both architectures frequently encounter optimization hurdles like vanishing or exploding gradients. This paper introduces DeepMR, a hybrid architecture designed to leverage the strengths of both paradigms while utilizing ReZero connections to ensure training stability and superior representation learning.
Methodology
The DeepMR architecture is composed of three primary stages: embedding, parallel representation learning, and weighted fusion.
Field Embedding and Positional Encoding
Input categorical features are first transformed into low-dimensional latent vectors through an embedding layer. To distinguish between different feature fields—a critical requirement for self-attention mechanisms—the model incorporates positional encodings, a technique adapted from Transformer architectures in Natural Language Processing.
The DeepMR Architecture
The core of the model consists of two parallel branches that process the embedded inputs simultaneously:
- ReZero DNN Component: This branch utilizes standard fully connected layers. However, it replaces traditional residual connections with ReZero (Residual with Zero initialization). In a ReZero layer, the output is computed as $x_{i+1} = x_i + \alpha_i \cdot F(x_i)$, where $\alpha$ is a learnable scalar initialized to zero. This allows the network to start as a simple linear identity mapping and gradually increase complexity, facilitating better gradient flow and faster convergence.
- ReZero Multi-head Self-Attention (MHSA) Component: This branch is dedicated to capturing multiplicative interactions. It also employs ReZero to stabilize the attention mechanism. The authors specifically found that a multiplicative variant of ReZero performed best for this component.
Weighted Combination
The final prediction $\hat{y}$ is a weighted sum of the outputs from the two branches, controlled by a balancing factor $\beta$: $$\hat{y} = \text{Combination}((1-\beta) \cdot \text{Output}_{DNN} + \beta \cdot \text{Output}_{MHSA})$$ This allows the model to adaptively prioritize additive or multiplicative interactions based on the specific characteristics of the dataset.
Experimental Evaluation
DeepMR was evaluated against several state-of-the-art baselines, including DeepFM, xDeepFM, and AutoInt+, using three real-world datasets: Frappe (app usage), MovieLens (movie tags), and LastFM (music tags). Performance was measured using Area Under the ROC Curve (AUC) and Logloss.
Key Results and Sensitivity Analysis
The experimental results demonstrate that DeepMR consistently achieves state-of-the-art performance. A critical finding was the impact of the ReZero connection on Logloss; for the LastFM dataset, incorporating ReZero reduced Logloss from 0.3141 to 0.2840, indicating significantly more calibrated probability estimates. The improvements over the best-performing baselines were found to be statistically significant at the 0.01 level.
As shown in the hyperparameter analysis, the optimal value for $\beta$ is dataset-dependent. For instance, some datasets benefit more from a higher reliance on the attention mechanism, while others favor the DNN branch. Furthermore, the model reaches peak performance with a relatively small number of layers (typically 1 to 5), suggesting that the ReZero-enhanced architecture is highly efficient at extracting meaningful features without requiring extreme depth.
Conclusion
DeepMR introduces a robust framework for CTR prediction by fusing DNNs and self-attention in a parallel architecture stabilized by ReZero connections. The model not only achieves superior predictive accuracy but also exhibits improved training stability and better-calibrated outputs. Future research will focus on automating the tuning of the $\beta$ parameter and exploring the integration of continuous numerical features and unstructured data types.
Summary updated