TL;DR
Existing generative models for Click-Through Rate (CTR) prediction suffer from an "asymmetric paradigm" flaw: they use generative objectives during training to learn better representations but revert to standard discriminative methods during online inference. This paper introduces SGCTR (Symmetric Masked Generative Paradigm), which bridges this gap by using the model's generative capabilities during inference to iteratively denoise and refine input features, leading to significant accuracy gains in both offline benchmarks and real-world online production.
What Problem Does This Paper Address?
Click-Through Rate (CTR) prediction is the backbone of modern recommender systems and online advertising. Traditionally, models use a discriminative paradigm—taking user and item features to output a binary classification score (click vs. no-click). Recently, researchers have integrated generative paradigms (like discrete diffusion or auto-regressive sequencing) during training to help the model capture complex feature dependencies and combat data sparsity.
However, a critical bottleneck remains: inference asymmetry. During online deployment, these generative models discard their generation capabilities and predict directly on raw, potentially noisy input samples. The authors tackle this limitation by asking: Can we use the model's generative power during inference to actively clean and optimize the input data before making a prediction?
Methods
SGCTR establishes a symmetric framework split into two distinct phases:
-
Training Phase (Discrete Diffusion): The model learns feature correlations by corrupting input samples with a binary mask token based on a noise scheduling function. To handle the massive search space caused by high-cardinality ID features, it utilizes a sampled softmax approximation, calculating loss over a random subset of negative samples.
-
Inference Phase (Iterative Refinement): Instead of a single-forward pass, SGCTR treats inference as an active, iterative denoising process:
-
Initialization: Stable user-side features are kept, while item-side and cross-features are initially masked.
- Feature Generation & Confidence Scoring: The model predicts the masked features in parallel. It then computes a confidence score for each prediction based on its cosine similarity to the original input feature.
- Feature Redefine and Mask: Features with the lowest confidence scores are re-masked for the next round, while highly confident features are retained and scaled by their confidence weights.
- Prediction: After $T$ discrete steps, the fully refined sample is fed into the final scoring function to predict the CTR.
To make this practical for low-latency online deployment, the authors use Key-Value (KV) caching to avoid re-computing stable features, keeping the overall computational complexity at $O(n)$.
Datasets
The authors evaluated SGCTR on four large-scale datasets:
- Criteo: A canonical public benchmark containing 45 million impressions, 13 continuous features, and 26 categorical features.
- Avazu: 40 million chronologically ordered ad click logs across 23 feature fields.
- Malware: A Kaggle competition dataset with 8.9 million impressions and 81 feature fields.
- Industrial Dataset: A massive proprietary dataset gathered from a real-world e-commerce platform's online advertising system, containing 782 million impressions across 72 feature fields.
Key Insights and Ideas
- Symmetry Unlocks Potential: Generative models shouldn't just be passive feature extractors during training. Allowing the model to "infer as it trains" lets it actively correct noisy or missing data at runtime.
- The Power of Concave Scheduling: When deciding how many features to mask at each step, concave functions (like Cosine or Square schedules) perform best. They maintain a high mask ratio early on, forcing the model to focus on refining a small core of highly confident features before filling in the rest.
- More Iterations Aren't Always Better: Performance peaks at a moderate number of steps ($T=5$ to $8$) and declines if the loop runs too long. Too many steps impose overly strict constraints on feature selection and degrade the final prediction score.
Key Results
SGCTR consistently outperformed both traditional discriminative baselines (like DeepFM, DCN, and HSTU) and state-of-the-art generative baselines (DGenCTR) across all datasets.
- Offline Evaluation: SGCTR achieved statistically significant improvements in Area Under the ROC Curve (AUC) across all four benchmarks, proving that inference-phase feature refinement works.
- Online Production Lift: In a 10-day online A/B test on a large-scale e-commerce platform, SGCTR achieved a 2.1% relative improvement in CTR compared to a production-grade DGenCTR baseline.
Limitations and Future Work
The primary limitation acknowledged by the authors involves generating high-cardinality ID features from scratch. In an ablation study variant where the model completely replaced original features with purely generated ones (SGCTR(GenFea)), performance dropped significantly. Because of this, the authors had to compromise by using a redefinition/weighting method rather than pure generation.
Future work will focus on enhancing the generative fidelity of the model so it can reliably synthesize complex, high-dimensional ID features from scratch without relying on original inputs as a crutch.
Summary updated