Deep Match to Rank Model for Personalized Click-Through Rate Prediction

Zequn Lyu, Yu Dong, Chengfu Huo, Weijun RenAAAI 2020 · DOI

Introduction

In modern recommendation systems, the pipeline is typically bifurcated into two distinct stages: Matching and Ranking. Matching involves retrieving a candidate set of a few hundred items from a massive pool of millions, while Ranking focuses on predicting the precise probability of a click (CTR) for these candidates.

While traditional matching methods like Collaborative Filtering excel at capturing user-item "relevance," modern deep-learning-based ranking models often prioritize complex feature interactions or generalized user representations. This shift can neglect the direct, personalized relevance signal—the "intensity of a user's preference"—which is critical for high-accuracy CTR prediction. The Deep Match to Rank (DMR) model addresses this gap by explicitly modeling user-to-item relevance within a ranking framework.

The DMR Architecture

DMR introduces a hybrid architecture that integrates matching principles into a predictive ranking model. The architecture consists of two specialized sub-networks and an auxiliary task designed to capture multi-faceted relevance.

The architecture of the Deep Match to Rank (DMR) model, highlighting the User-to-Item and Item-to-Item networks.

Core Components

  1. User-to-Item (U2I) Network: This component calculates relevance by taking the inner product of a user representation (derived from historical behavior) and a target item representation.
  2. Auxiliary Match Network: To ensure the U2I network learns meaningful relevance, an auxiliary task is introduced: predicting the next item a user will click based on their history. This task uses negative sampling for efficiency and serves as a supervisor to "push" the model toward higher inner products for relevant items.
  3. Item-to-Item (I2I) Network: This sub-network employs an attention mechanism to calculate similarities between previously interacted items and the target item. These similarities are aggregated to form a second relevance feature.
  4. Positional Encoding: To account for the fact that recent behaviors are often more predictive, positional encoding is integrated into the attention mechanism, allowing the model to weight sequence order.
  5. Joint Training: The model is optimized using a combined loss function: the standard target CTR loss plus the auxiliary matching loss.

Experimental Evaluation

The authors evaluated DMR on two large-scale datasets: the public Alimama dataset (26 million logs) and a proprietary Industrial Dataset from Alibaba (1.18 billion examples).

Performance Metrics

DMR was compared against state-of-the-art models including Wide&Deep, Deep Interest Network (DIN), and Deep Interest Evolution Network (DIEN).

  • Offline Performance: On the Alimama dataset, DMR achieved an AUC of 0.6447, outperforming DIEN (0.6420). On the industrial dataset, it reached an AUC of 0.6921.
  • Online Impact: In live A/B testing at Alibaba, DMR delivered a 5.5% improvement in CTR and a 12.8% increase in clicks per user compared to the previous production model.
Comparison of relevance values for positive and negative examples in the U2I and I2I networks across datasets.

The effectiveness of the relevance modeling is visualized in the difference between relevance scores for positive (clicked) and negative (unclicked) examples, as shown in the figure above.

Key Insights

  • Explicit Relevance: By making relevance an explicit feature rather than an implicit latent property, DMR provides a stronger signal to the final prediction layers.
  • Auxiliary Supervision: The auxiliary matching task is vital. Without it, the sparse click signal is often insufficient to supervise the complex relationship between users and items in the U2I network.
  • Temporal Dynamics: The success of positional encoding confirms that "recency" is a dominant factor in user behavior, with the model learning to prioritize recent actions.

Conclusion

While DMR significantly improves CTR prediction, there are avenues for further refinement. Integrating additional contextual features (e.g., dwell time or behavior type) into the attention mechanism could provide even finer-grained modeling. Furthermore, while DMR avoids Recurrent Neural Networks (RNNs) to maintain low latency in production, exploring efficient sequential modeling techniques remains a priority. Finally, the infrastructure requirements for training on billions of items with auxiliary tasks present a scalability challenge for smaller-scale deployments.

Summary updated

BibTeX
@inproceedings{lyu2020deep,
  author = {Zequn Lyu and Yu Dong and Chengfu Huo and Weijun Ren},
  title = {{Deep Match to Rank Model for Personalized Click-Through Rate Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5346},
}
Figure