Deep Multi-Representation Model for Click-Through Rate Prediction

Shereen Elsayed, Lars Schmidt-Thieme

TL;DR

DeepMR is a novel architecture for Click-Through Rate (CTR) prediction that combines the strengths of Deep Neural Networks (DNNs) and Multi-head Self-Attention in a parallel "mixture of experts" style. By incorporating ReZero (Residual with Zero initialization) connections, the model achieves state-of-the-art performance and more stable training across multiple real-world datasets.

What Problem Does This Paper Address?

In the world of online advertising and recommender systems, predicting whether a user will click an item (CTR prediction) is a multi-billion dollar problem. The challenge lies in capturing "feature interactions"—the complex ways different data fields (like user age, time of day, and item category) relate to one another.

Existing models usually fall into two camps:

  1. DNN-based models: Excellent at capturing high-order, non-linear additive interactions.
  2. Attention-based models: Excellent at capturing multiplicative relations and similarities between different fields.

However, both have limitations. DNNs often fail to capture multiplicative relations effectively, and both types of models can suffer from training instabilities (like vanishing or exploding gradients) as they get deeper. The authors seek to bridge this gap by creating a model that uses both representations simultaneously while improving training stability.

Methods

The authors propose DeepMR (Deep Multi-Representation), which consists of three main building blocks:

  • Field Embedding with Positional Encoding: Categorical input fields are mapped to latent vectors. Crucially, the authors add positional encodings to help the model distinguish between different input fields, a technique borrowed from Natural Language Processing.
  • ReZero DNN Component: This branch uses standard fully connected layers but replaces traditional residual connections with ReZero. In ReZero, each residual layer has a learnable scalar weight ($\alpha$) initialized to zero. This allows the model to behave like a shallow, linear network at the start of training and gradually "evolve" into a complex deep network, leading to faster convergence and better signal propagation.
  • ReZero Multi-head Self-Attention (MHSA) Component: This branch captures multiplicative interactions. Like the DNN branch, it utilizes ReZero connections to refine the representations.
  • Weighted Combination: The outputs of both branches are combined using a weighted sum controlled by a factor $\beta$. This allows the model to tune how much it relies on the DNN vs. the Attention mechanism for a specific dataset.

Datasets

The model was evaluated on three public, real-world datasets commonly used in recommendation research:

  • Frappe: A mobile app usage dataset (context-aware recommendations).
  • MovieLens: A dataset focused on personalized movie tag recommendations.
  • LastFM: A dataset for personalized artist-tag recommendations.

The authors used standard evaluation metrics: AUC (Area Under the ROC Curve) for classification accuracy and Logloss to measure the "certainty" of the predictions.

Key Insights and Ideas

  • The Power of ReZero in CTR: While ReZero was originally designed for very deep Transformers in NLP, this paper demonstrates it is highly effective for CTR models. It significantly improves Logloss, suggesting that it helps the model produce more calibrated and confident probability estimates.
  • Multiplicative vs. Additive Residuals: The authors found that for the self-attention block, a multiplicative ReZero connection (scaling the output of the attention layer before adding it back) performed better than a standard additive one.
  • Parallelism Matters: By running the DNN and Attention blocks in parallel rather than stacking them, the model can extract two distinct "views" of the data, which are then fused at the final layer.
  • Dataset Sensitivity: The optimal balance ($\beta$) between the two branches varies by dataset. For some datasets, the Attention branch is more critical; for others, the DNN provides the heavy lifting.

Key Results

  • State-of-the-Art Performance: DeepMR consistently outperformed several high-profile baselines, including DeepFM, xDeepFM, AutoInt+, and DCN-Mix.
  • Ablation Success: Removing the ReZero connections led to a significant drop in performance, particularly in Logloss. For example, on the LastFM dataset, adding ReZero dropped the Logloss from 0.3141 to 0.2840.
  • Statistical Significance: The improvements over the best-performing baselines were found to be statistically significant at the 0.01 level.
  • Layer Efficiency: The authors found that thanks to ReZero, the model achieved peak performance with relatively few layers (1 to 5, depending on the dataset), avoiding the unnecessary complexity often found in deep CTR models.

Limitations and Future Work

  • Hyperparameter Tuning: The balancing factor $\beta$ is highly sensitive. While the authors experimented with learning $\beta$ automatically during training, it didn't always outperform a manually tuned, fixed $\beta$. Finding a more robust way to automate this balance is a potential area for improvement.
  • Computational Complexity: Maintaining two separate, complex branches (DNN and MHSA) increases the number of parameters and the computational cost during inference compared to simpler single-branch models.
  • Feature Scope: The current work focuses primarily on categorical/field-based data. Future iterations could explore how this multi-representation approach handles continuous numerical features or unstructured data like text or images within the same framework.

Summary updated

Abstract

Click-Through Rate prediction (CTR) is a crucial task for online advertising and recommender systems. Therefore, it has gained considerable attention in the past few years as it highly affects the revenue of several commercial platforms and online systems. The primary purpose of recent research emphasizes obtaining meaningful and powerful representations through mining low and high-feature interactions using various components such as Deep Neural Networks (DNN), CrossNets, or transformer blocks. However, models utilizing one representation for the input fields in each instance restrict the model's predictive power. Other models tend to be overly complicated to reach high input data expressiveness and predictive power. In this work, we propose a simple yet effective Deep Multi-Representation model (DeepMR) that is capable of learning informative representations by jointly training a mixture of two powerful feature representation learning components, namely DNNs and multi-head self-attentions. Furthermore, DeepMR integrates the novel residual with zero initialization (ReZero) connections to the DNN and the multi-head self-attention components for learning superior input representations. Experiments on three real-world datasets show that the proposed model significantly outperforms all state-of-the-art models with a relative improvement of up to 16.6% in the task of click-through rate prediction. Our implementation code and datasets are available here https://github.com/Shereen-Elsayed/DeepMR.
BibTeX
@inproceedings{elsayed2023deep,
  author = {Shereen Elsayed and Lars Schmidt-Thieme},
  title = {{Deep Multi-Representation Model for Click-Through Rate Prediction}},
  booktitle = {IJCNN},
  year = {2023},
  doi = {10.1109/ijcnn54540.2023.10191755},
}

Loading related content...

Figure