TL;DR
This paper introduces the Deep Interest Network (DIN), a novel deep learning model for click-through rate (CTR) prediction in online advertising. DIN addresses the limitation of fixed-length user interest representations in existing models by adaptively learning user interests based on the specific ad being considered, leading to improved prediction accuracy. The authors also propose two practical techniques for training large-scale industrial deep networks: mini-batch aware regularization and a data-adaptive activation function.
What Problem Does This Paper Address?
Click-through rate (CTR) prediction is crucial for online advertising systems, directly impacting revenue. Traditional deep learning approaches for CTR prediction, often termed "Embedding&MLP," map sparse input features into low-dimensional embeddings, then transform them into fixed-length vectors. While effective, this fixed-length representation struggles to capture the diverse interests of users, especially those with rich historical behavior data. For instance, a user interested in both electronics and fashion might have their diverse interests compressed into a single vector, limiting the model's ability to accurately predict their interest in a specific ad. This paper aims to overcome this bottleneck by creating a more dynamic and context-aware representation of user interests.
Methods
The core innovation is the Deep Interest Network (DIN). DIN introduces a local activation unit that adaptively learns a user's interest representation with respect to a candidate ad. Instead of a single fixed-length vector for all ads, DIN calculates a weighted sum of user behavior embeddings, where the weights are determined by the relevance of each behavior to the current ad. This allows the representation to vary for different ads, better reflecting the user's context-specific interests.
To facilitate the training of massive industrial deep networks, two novel techniques are proposed:
- Mini-batch Aware Regularization: This technique efficiently applies L2 regularization by only considering parameters of features present in the current mini-batch, significantly reducing computational overhead compared to traditional methods that require calculating norms over all parameters.
- Data Adaptive Activation Function (Dice): This function generalizes PReLU by adaptively adjusting the rectified point based on the input data's distribution (mean and variance), leading to improved training stability and performance, especially for sparse features.
Datasets
The paper evaluates its methods on three datasets:
- Amazon Dataset (Electronics subset): A public dataset containing user reviews and metadata, used for evaluating models on user behavior prediction tasks.
- MovieLens Dataset: A public dataset commonly used for recommendation systems, adapted for CTR prediction by transforming ratings into binary labels.
- Alibaba Dataset: A large-scale, real-world dataset collected from Alibaba's online display advertising system, containing over 2 billion samples, which serves as a crucial benchmark for industrial-scale performance.
Key Insights and Ideas
- User interests are diverse and context-dependent: The paper strongly emphasizes that a user's interests are not monolithic. What drives a click on one ad might be entirely different from what drives a click on another.
- Fixed-length representations are a bottleneck: Compressing all user interests into a single, fixed-size vector inherently loses information and limits the model's ability to capture nuanced preferences.
- Local activation is key: By focusing on the relevance of historical behaviors to a specific ad, DIN can dynamically construct a more accurate representation of user interest for that particular context.
- Practical training techniques are essential for scale: The proposed mini-batch aware regularization and data-adaptive activation function are critical for making deep learning models feasible and effective in real-world, large-scale industrial settings.
Key Results
- Superior CTR Prediction: DIN consistently outperformed state-of-the-art models, including BaseModel, Wide&Deep, PNN, and DeepFM, on all tested datasets. On the Alibaba dataset, DIN achieved a significant absolute AUC gain of 0.0059 and 6.08% RelaImpr over the BaseModel.
- Improved Training Efficiency: The mini-batch aware regularization significantly improved performance on large-scale datasets with fine-grained features, preventing overfitting and achieving higher AUC compared to other regularization methods.
- Enhanced Model Performance: The Dice activation function further boosted DIN's performance, demonstrating its effectiveness in handling diverse data distributions.
- Real-world Deployment: DIN, along with its proposed training techniques, has been successfully deployed in Alibaba's online display advertising system, serving main traffic and contributing to significant improvements in CTR and RPM.
Limitations and Future Work
- Sequential Behavior Modeling: The authors experimented with LSTM for modeling user historical behavior data sequentially but found no improvement. They suggest that designing special structures to model such data in a sequence way is a direction for future research, acknowledging that the sequence of user behaviors can be noisy and contain multiple concurrent interests.
- Generalizability of Techniques: While the paper focuses on CTR prediction, the proposed training techniques (mini-batch aware regularization and Dice activation) are presented as generalizable to other industrial deep learning tasks involving large-scale sparse features.
Summary updated