Modeling Long-term User Behaviors with Diffusion-driven Multi-interest Network for CTR Prediction

Weijiang Lai, Beihong Jin, Yapeng Zhang, Yiyuan Zheng, Rui Zhao, Jian Dong, Jun Lei, Xingxing Wang

TL;DR

This paper introduces DiffuMIN, a novel network for Click-Through Rate (CTR) prediction that models long-term user behavior more effectively. By combining a target-oriented multi-interest extraction method with a diffusion-based generative approach, DiffuMIN captures a richer and more diverse user interest space, leading to improved prediction accuracy and real-world performance gains.

What Problem Does This Paper Address?

Predicting whether a user will click on an item (CTR prediction) is fundamental to online advertising and recommendation systems. While modeling users' long-term behavior is known to boost CTR prediction accuracy, existing methods face challenges. The sheer volume of historical data and the presence of noisy interactions make it difficult to efficiently and effectively extract meaningful user interests. Current two-stage models, which first filter behaviors and then model them, often discard valuable information, leading to an incomplete representation of a user's diverse interests. This limits their ability to fully capture the user's latent interest space.

Methods

DiffuMIN tackles these challenges with a multi-stage approach:

  1. Target-Oriented Multi-Interest Extraction (OMIE): This novel first stage aims to disentangle and extract multiple user interests from long-term behaviors. It achieves this by:
  2. Orthogonal Decomposition: Decomposing the target item's embedding into orthogonal "interest channels." These channels act as distinct lenses through which user behaviors are viewed.
  3. Behavior Routing & Filtering: Behaviors are routed to the most relevant interest channel based on their score. Then, only the top-performing behaviors within each channel are retained, reducing redundancy and noise.
  4. Interest Aggregation: The filtered behaviors within each channel are aggregated (e.g., using mean pooling) to form distinct "aggregated interests."
  5. Diffusion Multi-Interest Generator (DMIG): Inspired by generative modeling, this stage expands the user's interest space. Instead of relying solely on observed behaviors, it uses a diffusion model to generate "augmented interests."
    • Guided Diffusion: The diffusion process is guided by both "contextual interests" (derived from the user's recent interactions) and the "interest channels" from the OMIE stage. This ensures the generated interests are personalized and target-oriented.
  6. Perturbed Starting Points: Unlike traditional diffusion models that start from Gaussian noise, DiffuMIN starts with perturbed user interests, enhancing personalization and simplifying generation.
  7. Contrastive Multi-Interest Calibrator (CMIC): To ensure the generated augmented interests truly reflect user preferences and to improve distinguishability, contrastive learning is employed. It pulls augmented interests closer to the user's aggregated interests (positive samples) while pushing them away from other users' interests (negative samples).
  8. Prediction Layer: Finally, the aggregated interests, augmented interests, and short-term user interests are fed into a Multi-Layer Perceptron (MLP) for the final CTR prediction.

Datasets

The paper evaluates DiffuMIN on three datasets:

  • Ele.me: A public dataset from an online recommender system, containing 30-day user behavior logs.
  • Alibaba: A public display advertising CTR prediction dataset, covering user shopping behavior over 22 days.
  • Industry: A large-scale industrial dataset with over 84 million users and comprehensive behavior records from the past year.

Key Insights and Ideas

  • Decomposing the Target is Key: The idea of using orthogonal "interest channels" derived from the target item's embedding is a novel way to segment and analyze user behaviors from multiple perspectives. This moves beyond simply looking at behavior-to-behavior similarity.
  • Generative Augmentation for Latent Interests: Diffusion models are powerful for capturing complex data distributions. Applying them to generate "augmented interests" effectively expands the user's interest space beyond what's directly observable, potentially uncovering latent preferences.
  • Guided Generation Matters: Simply generating interests isn't enough; guiding the diffusion process with contextual user information and the established interest channels makes the generated interests more relevant and personalized.
  • Contrastive Learning for Robustness: Using contrastive learning to align generated interests with actual preferences and differentiate them from other users' interests is crucial for improving the model's overall robustness and generalization.

Key Results

  • Superior Offline Performance: DiffuMIN consistently outperforms various state-of-the-art models (including DIN, CAN, SIM, ETA, SDIM, TWIN, TWIN-V2) across all three datasets in terms of AUC. For example, on the Industry dataset, DiffuMIN achieved a 5.80% relative improvement over DIN(S).
  • Significant Online Gains: In a 7-day online A/B test, DiffuMIN led to a 1.52% increase in CTR and a 1.10% increase in CPM, demonstrating its practical effectiveness.
  • Module Contributions: Ablation studies confirm that each component of DiffuMIN (OMIE, DMIG, CMIC) contributes significantly to its performance. Removing any of these modules leads to a noticeable drop in AUC.
  • Efficiency: Despite its complexity, DiffuMIN's time complexity is comparable to existing long-term behavior modeling methods, making it suitable for online deployment.

Limitations and Future Work

  • Computational Cost: While efficient, diffusion models inherently involve iterative processes, which can still be computationally intensive. The paper mentions that inference time increased slightly from 33ms to 35ms in the online test.
  • Hyperparameter Tuning: The performance of DiffuMIN is sensitive to hyperparameter choices (e.g., number of channels, diffusion steps, loss weights), requiring careful tuning.
  • Future Work: The authors suggest exploring more advanced diffusion techniques and further investigating how to leverage diffusion models for other aspects of CTR prediction or recommendation tasks. The paper also hints at the potential for further exploration of user interest spaces through generative methods.

Summary updated

Abstract

CTR (Click-Through Rate) prediction, crucial for recommender systems and online advertising, etc., has been confirmed to benefit from modeling long-term user behaviors. Nonetheless, the vast number of behaviors and complexity of noise interference pose challenges to prediction efficiency and effectiveness. Recent solutions have evolved from single-stage models to two-stage models. However, current two-stage models often filter out significant information, resulting in an inability to capture diverse user interests and build the complete latent space of user interests. Inspired by multi-interest and generative modeling, we propose DiffuMIN (Diffusion-driven Multi-Interest Network) to model long-term user behaviors and thoroughly explore the user interest space. Specifically, we propose a target-oriented multi-interest extraction method that begins by orthogonally decomposing the target to obtain interest channels. This is followed by modeling the relationships between interest channels and user behaviors to disentangle and extract multiple user interests. We then adopt a diffusion module guided by contextual interests and interest channels, which anchor users' personalized and target-oriented interest types, enabling the generation of augmented interests that align with the latent spaces of user interests, thereby further exploring restricted interest space. Finally, we leverage contrastive learning to ensure that the generated augmented interests align with users' genuine preferences. Extensive offline experiments are conducted on two public datasets and one industrial dataset, yielding results that demonstrate the superiority of DiffuMIN. Moreover, DiffuMIN increased CTR by 1.52% and CPM by 1.10% in online A/B testing. Our source code is available at https://github.com/laiweijiang/DiffuMIN.
BibTeX
@misc{lai2025modeling,
  author = {Weijiang Lai and Beihong Jin and Yapeng Zhang and Yiyuan Zheng and Rui Zhao and Jian Dong and Jun Lei and Xingxing Wang},
  title = {{Modeling Long-term User Behaviors with Diffusion-driven Multi-interest Network for CTR Prediction}},
  year = {2025},
  eprint = {2508.15311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.15311},
}
Figure