Deep Interest Network for Click-Through Rate Prediction

Guorui Zhou, Chengru Song, Xiaoqiang Zhu, Xiao Ma, Yanghui Yan, Xingya Dai, Han Zhu, Junqi Jin, Han Li, Kun GaiarXiv 2017 · OpenReview

TL;DR

This paper introduces the Deep Interest Network (DIN), a novel deep learning model for click-through rate (CTR) prediction in online advertising. DIN addresses the limitation of fixed-length user interest representations in existing models by adaptively learning user interests based on the specific ad being considered, leading to improved prediction accuracy. The authors also propose two practical techniques for training large-scale industrial deep networks: mini-batch aware regularization and a data-adaptive activation function.

What Problem Does This Paper Address?

Click-through rate (CTR) prediction is crucial for online advertising systems, directly impacting revenue. Traditional deep learning approaches for CTR prediction, often termed "Embedding&MLP," map sparse input features into low-dimensional embeddings, then transform them into fixed-length vectors. While effective, this fixed-length representation struggles to capture the diverse interests of users, especially those with rich historical behavior data. For instance, a user interested in both electronics and fashion might have their diverse interests compressed into a single vector, limiting the model's ability to accurately predict their interest in a specific ad. This paper aims to overcome this bottleneck by creating a more dynamic and context-aware representation of user interests.

Methods

The core innovation is the Deep Interest Network (DIN). DIN introduces a local activation unit that adaptively learns a user's interest representation with respect to a candidate ad. Instead of a single fixed-length vector for all ads, DIN calculates a weighted sum of user behavior embeddings, where the weights are determined by the relevance of each behavior to the current ad. This allows the representation to vary for different ads, better reflecting the user's context-specific interests.

To facilitate the training of massive industrial deep networks, two novel techniques are proposed:

  1. Mini-batch Aware Regularization: This technique efficiently applies L2 regularization by only considering parameters of features present in the current mini-batch, significantly reducing computational overhead compared to traditional methods that require calculating norms over all parameters.
  2. Data Adaptive Activation Function (Dice): This function generalizes PReLU by adaptively adjusting the rectified point based on the input data's distribution (mean and variance), leading to improved training stability and performance, especially for sparse features.

Datasets

The paper evaluates its methods on three datasets:

  • Amazon Dataset (Electronics subset): A public dataset containing user reviews and metadata, used for evaluating models on user behavior prediction tasks.
  • MovieLens Dataset: A public dataset commonly used for recommendation systems, adapted for CTR prediction by transforming ratings into binary labels.
  • Alibaba Dataset: A large-scale, real-world dataset collected from Alibaba's online display advertising system, containing over 2 billion samples, which serves as a crucial benchmark for industrial-scale performance.

Key Insights and Ideas

  • User interests are diverse and context-dependent: The paper strongly emphasizes that a user's interests are not monolithic. What drives a click on one ad might be entirely different from what drives a click on another.
  • Fixed-length representations are a bottleneck: Compressing all user interests into a single, fixed-size vector inherently loses information and limits the model's ability to capture nuanced preferences.
  • Local activation is key: By focusing on the relevance of historical behaviors to a specific ad, DIN can dynamically construct a more accurate representation of user interest for that particular context.
  • Practical training techniques are essential for scale: The proposed mini-batch aware regularization and data-adaptive activation function are critical for making deep learning models feasible and effective in real-world, large-scale industrial settings.

Key Results

  • Superior CTR Prediction: DIN consistently outperformed state-of-the-art models, including BaseModel, Wide&Deep, PNN, and DeepFM, on all tested datasets. On the Alibaba dataset, DIN achieved a significant absolute AUC gain of 0.0059 and 6.08% RelaImpr over the BaseModel.
  • Improved Training Efficiency: The mini-batch aware regularization significantly improved performance on large-scale datasets with fine-grained features, preventing overfitting and achieving higher AUC compared to other regularization methods.
  • Enhanced Model Performance: The Dice activation function further boosted DIN's performance, demonstrating its effectiveness in handling diverse data distributions.
  • Real-world Deployment: DIN, along with its proposed training techniques, has been successfully deployed in Alibaba's online display advertising system, serving main traffic and contributing to significant improvements in CTR and RPM.

Limitations and Future Work

  • Sequential Behavior Modeling: The authors experimented with LSTM for modeling user historical behavior data sequentially but found no improvement. They suggest that designing special structures to model such data in a sequence way is a direction for future research, acknowledging that the sequence of user behaviors can be noisy and contain multiple concurrent interests.
  • Generalizability of Techniques: While the paper focuses on CTR prediction, the proposed training techniques (mini-batch aware regularization and Dice activation) are presented as generalizable to other industrial deep learning tasks involving large-scale sparse features.

Summary updated

Abstract

Click-through rate prediction is an essential task in industrial applications, such as online advertising. Recently deep learning based models have been proposed, which follow a similar Embedding\&MLP paradigm. In these methods large scale sparse input features are first mapped into low dimensional embedding vectors, and then transformed into fixed-length vectors in a group-wise manner, finally concatenated together to fed into a multilayer perceptron (MLP) to learn the nonlinear relations among features. In this way, user features are compressed into a fixed-length representation vector, in regardless of what candidate ads are. The use of fixed-length vector will be a bottleneck, which brings difficulty for Embedding\&MLP methods to capture user's diverse interests effectively from rich historical behaviors. In this paper, we propose a novel model: Deep Interest Network (DIN) which tackles this challenge by designing a local activation unit to adaptively learn the representation of user interests from historical behaviors with respect to a certain ad. This representation vector varies over different ads, improving the expressive ability of model greatly. Besides, we develop two techniques: mini-batch aware regularization and data adaptive activation function which can help training industrial deep networks with hundreds of millions of parameters. Experiments on two public datasets as well as an Alibaba real production dataset with over 2 billion samples demonstrate the effectiveness of proposed approaches, which achieve superior performance compared with state-of-the-art methods. DIN now has been successfully deployed in the online display advertising system in Alibaba, serving the main traffic.
BibTeX
@misc{zhou2017deep,
  author = {Guorui Zhou and Chengru Song and Xiaoqiang Zhu and Xiao Ma and Yanghui Yan and Xingya Dai and Han Zhu and Junqi Jin and Han Li and Kun Gai},
  title = {{Deep Interest Network for Click-Through Rate Prediction}},
  year = {2017},
  eprint = {1706.06978},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.06978},
}
Figure