Abstract
The task of predicting conversion rates (CVR) lies at the heart of online advertising systems aiming to optimize bids to meet advertiser performance requirements. Even with the recent rise of deep neural networks, these predictions are often made by factorization machines (FM), especially in commercial settings where inference latency is key. These models are trained using the logistic regression framework on labeled tabular data formed from past user activity that is relevant to the task at hand. Many advertisers only care about click-attributed conversions. A major challenge in training models that predict conversions-given-clicks comes from data sparsity - clicks are rare, conversions attributed to clicks are even rarer. However, mitigating sparsity by adding conversions that are not click-attributed to the training set impairs model calibration. Since calibration is critical to achieving advertiser goals, this is infeasible. In this work we use the well-known idea of self-supervised pre-training, and use an auxiliary auto-encoder model trained on all conversion events, both click-attributed and not, as a feature extractor to enrich the main CVR prediction model. Since the main model does not train on non click-attributed conversions, this does not impair calibration. We adapt the basic self-supervised pre-training idea to our online advertising setup by using a loss function designed for tabular data, facilitating continual learning by ensuring auto-encoder stability, and incorporating a neural network into a large-scale real-time ad auction that ranks tens of thousands of ads, under strict latency constraints, and without incurring a major engineering cost. We show improvements both offline, during training, and in an online A/B test. Following its success in A/B tests, our solution is now fully deployed to the Yahoo native advertising system.
Introduction: The Sparsity and Calibration Dilemma
In the competitive landscape of online advertising, predicting the Conversion Rate (CVR)—the probability that a user will perform a desired action, such as a purchase, after clicking an ad—is critical. Accurate pCVR (predicted CVR) values allow platforms to set optimal bids that balance advertiser return on investment (ROI) with platform revenue.
However, CVR models suffer from extreme data sparsity. While impressions are plentiful, clicks are rare, and conversions are even rarer. A common pitfall is attempting to solve this by training on "view-through" conversions (conversions that occur after an impression but without a click). While this increases data volume, it introduces a "calibration crisis." Because the model is trained on non-click data, its predictions for click-based conversions become biased. In programmatic auctions where bid = pCVR × target CPA, even small calibration errors lead to massive inefficiencies and missed advertiser goals.
Proposed Method: Self-Supervised Feature Extraction
To address sparsity without sacrificing calibration, the researchers developed a two-stage self-supervised framework that works within the strict latency and incremental learning requirements of a real-time ad auction.
- Self-Supervised Auto-encoder: An auxiliary model is trained on all conversion events (both click-attributed and not). This model acts as a feature extractor, learning to compress high-dimensional tabular features (user, context, ad) into a low-dimensional "Code." Since the auto-encoder learns feature structures rather than predicting conversion labels, it can utilize "dirty" or unlabeled data without biasing the final prediction.
- Integration via Random Fourier Features (RFF): The "Code" is fed into the main CVR model (typically a Factorization Machine). To bridge the gap between the non-linear neural code and the linear FM, the authors use Random Fourier Features. This acts as a projector that maps the code into a space where linear interactions can effectively capture complex patterns.
- Latency-Aware Inference: To maintain millisecond-level response times, the system employs taxonomy-based inference. Instead of calculating codes for every individual ad, the model uses high-level categories (e.g., "Finance" or "Automotive"), allowing the neural network outputs to be cached and reused across many ads in the same auction.
Experimental Results
The framework was tested on Yahoo’s native advertising marketplace, which handles billions of daily impressions and hundreds of millions of dollars in annual revenue.
- Offline Performance: Traditional methods of adding view-attributed data directly to the training set increased logloss (error) by 3-7%. In contrast, the self-supervised auto-encoder approach significantly improved predictive accuracy.
- Online Impact: In a large-scale A/B test on 50% of Yahoo's traffic, the model delivered a 0.67% total revenue lift and a 0.6% CPM lift.
- Advertiser Success: Campaigns using optimized bidding saw a 10.1% increase in spend, indicating that advertisers were winning more high-quality auctions and meeting their performance targets more effectively.
- Zero Latency Penalty: Thanks to the taxonomy-based caching strategy, there was no visible increase in auction latency despite adding a neural network to the pipeline.
Conclusion
By decoupling feature representation learning from label prediction, this research demonstrates a viable path to overcoming data sparsity in CVR models. This self-supervised approach provides the benefits of large-scale data utilization while maintaining the strict calibration and latency requirements of modern ad exchanges. The system is now fully deployed, contributing to significant annual revenue growth for the Yahoo native advertising system.
Summary updated