TL;DR
DeepMR is a novel architecture for Click-Through Rate (CTR) prediction that combines the strengths of Deep Neural Networks (DNNs) and Multi-head Self-Attention in a parallel "mixture of experts" style. By incorporating ReZero (Residual with Zero initialization) connections, the model achieves state-of-the-art performance and more stable training across multiple real-world datasets.
What Problem Does This Paper Address?
In the world of online advertising and recommender systems, predicting whether a user will click an item (CTR prediction) is a multi-billion dollar problem. The challenge lies in capturing "feature interactions"—the complex ways different data fields (like user age, time of day, and item category) relate to one another.
Existing models usually fall into two camps:
- DNN-based models: Excellent at capturing high-order, non-linear additive interactions.
- Attention-based models: Excellent at capturing multiplicative relations and similarities between different fields.
However, both have limitations. DNNs often fail to capture multiplicative relations effectively, and both types of models can suffer from training instabilities (like vanishing or exploding gradients) as they get deeper. The authors seek to bridge this gap by creating a model that uses both representations simultaneously while improving training stability.
Methods
The authors propose DeepMR (Deep Multi-Representation), which consists of three main building blocks:
- Field Embedding with Positional Encoding: Categorical input fields are mapped to latent vectors. Crucially, the authors add positional encodings to help the model distinguish between different input fields, a technique borrowed from Natural Language Processing.
- ReZero DNN Component: This branch uses standard fully connected layers but replaces traditional residual connections with ReZero. In ReZero, each residual layer has a learnable scalar weight ($\alpha$) initialized to zero. This allows the model to behave like a shallow, linear network at the start of training and gradually "evolve" into a complex deep network, leading to faster convergence and better signal propagation.
- ReZero Multi-head Self-Attention (MHSA) Component: This branch captures multiplicative interactions. Like the DNN branch, it utilizes ReZero connections to refine the representations.
- Weighted Combination: The outputs of both branches are combined using a weighted sum controlled by a factor $\beta$. This allows the model to tune how much it relies on the DNN vs. the Attention mechanism for a specific dataset.
Datasets
The model was evaluated on three public, real-world datasets commonly used in recommendation research:
- Frappe: A mobile app usage dataset (context-aware recommendations).
- MovieLens: A dataset focused on personalized movie tag recommendations.
- LastFM: A dataset for personalized artist-tag recommendations.
The authors used standard evaluation metrics: AUC (Area Under the ROC Curve) for classification accuracy and Logloss to measure the "certainty" of the predictions.
Key Insights and Ideas
- The Power of ReZero in CTR: While ReZero was originally designed for very deep Transformers in NLP, this paper demonstrates it is highly effective for CTR models. It significantly improves Logloss, suggesting that it helps the model produce more calibrated and confident probability estimates.
- Multiplicative vs. Additive Residuals: The authors found that for the self-attention block, a multiplicative ReZero connection (scaling the output of the attention layer before adding it back) performed better than a standard additive one.
- Parallelism Matters: By running the DNN and Attention blocks in parallel rather than stacking them, the model can extract two distinct "views" of the data, which are then fused at the final layer.
- Dataset Sensitivity: The optimal balance ($\beta$) between the two branches varies by dataset. For some datasets, the Attention branch is more critical; for others, the DNN provides the heavy lifting.
Key Results
- State-of-the-Art Performance: DeepMR consistently outperformed several high-profile baselines, including DeepFM, xDeepFM, AutoInt+, and DCN-Mix.
- Ablation Success: Removing the ReZero connections led to a significant drop in performance, particularly in Logloss. For example, on the LastFM dataset, adding ReZero dropped the Logloss from 0.3141 to 0.2840.
- Statistical Significance: The improvements over the best-performing baselines were found to be statistically significant at the 0.01 level.
- Layer Efficiency: The authors found that thanks to ReZero, the model achieved peak performance with relatively few layers (1 to 5, depending on the dataset), avoiding the unnecessary complexity often found in deep CTR models.
Limitations and Future Work
- Hyperparameter Tuning: The balancing factor $\beta$ is highly sensitive. While the authors experimented with learning $\beta$ automatically during training, it didn't always outperform a manually tuned, fixed $\beta$. Finding a more robust way to automate this balance is a potential area for improvement.
- Computational Complexity: Maintaining two separate, complex branches (DNN and MHSA) increases the number of parameters and the computational cost during inference compared to simpler single-branch models.
- Feature Scope: The current work focuses primarily on categorical/field-based data. Future iterations could explore how this multi-representation approach handles continuous numerical features or unstructured data like text or images within the same framework.
Summary updated