Predicting whether a user will actually buy something after clicking an ad—the Conversion Rate (CVR)—is the "holy grail" of online advertising. If your prediction is off, advertisers either overpay or miss out on customers, and the platform loses money.
In a recent paper, researchers from Yahoo Research detail how they successfully deployed self-supervised pre-training to solve the chronic data sparsity of CVR models, resulting in millions of dollars in annual revenue growth.
TL;DR
Yahoo researchers used an auxiliary auto-encoder trained on all conversion events (even those not directly caused by a click) to create "feature codes" that enrich their main CVR model. This self-supervised approach bypasses the "data sparsity" problem without ruining model calibration, leading to a 0.67% revenue lift in live production.
What Problem Does This Paper Address?
The core challenge in CVR prediction is data sparsity. While billions of ads are shown, clicks are rare, and conversions (purchases/sign-ups) are even rarer.
To get more data, you might be tempted to train your model on "view-through" conversions (conversions that happen after a user sees an ad but doesn't click). However, this creates a calibration crisis. In programmatic advertising, the bid is often calculated as:
bid = pCVR (predicted CVR) × tCPA (target cost per acquisition)
If you train on non-click data, your model becomes biased. It might predict a high conversion probability, causing the advertiser to overbid for a click that was unlikely to convert. Conversely, under-prediction means losing auctions. Most CVR models must be "calibrated"—meaning the average prediction must match the actual observed rate—and adding "unlabeled" or loosely related data usually breaks that calibration.
Methods
The authors propose a two-stage, self-supervised framework that works within the strict latency and incremental learning requirements of a real-time ad auction.
- The Auto-encoder (The Feature Extractor): They train an auto-encoder on all conversion data (categorical tabular data). The encoder compresses user, context, and ad features into a low-dimensional "code." Because the auto-encoder is just learning to reconstruct features, it can use all available data without biasing the final CVR prediction.
- Random Fourier Features (RFFs): To help the downstream linear model (a Factorization Machine) understand the non-linear "code" from the encoder, they apply RFFs. This acts as a "projector" that maps the code into a space where a simple linear function can differentiate between positive and negative samples.
- Taxonomy-based Inference: To meet millisecond latency constraints, the auto-encoder doesn't look at individual ad IDs. Instead, it uses high-level "taxonomy" features (e.g., "Finance" or "Tourism"). This allows the system to calculate the expensive neural network parts once per category and cache them, rather than calculating them for every single ad in an auction.
- Continual Learning: Unlike static models, this system trains incrementally on an infinite stream of data. The authors emphasize "stability"—ensuring the encoder’s output doesn't shift wildly between training intervals, which would confuse the main CVR model.
Datasets
The study was conducted using Yahoo’s native advertising marketplace data:
- Scale: Billions of daily impressions and hundreds of millions of dollars in annual revenue.
- Features: Tabular categorical data including user demographics, context (page content), and ad metadata.
- Evaluation: A month of offline logs for reconstruction and logloss tests, followed by a massive online A/B test on 50% of Yahoo's native traffic.
Key Insights and Ideas
- Calibration via Separation: The "aha moment" is that you can use "dirty" or "unlabeled" data to learn the structure of a conversion (via the auto-encoder) without using it to set the target for the prediction. This provides the benefits of Big Data without the bias of "bad" labels.
- Stability = Generalization: In online learning, if a model generalizes well to new data, its gradients remain small, and its parameters stay stable. The researchers found a direct linear correlation between a model's stability and its predictive power.
- Over-parameterization is Okay: Despite classical warnings against overfitting, the authors found that increasing the complexity of the encoder (making it "over-parameterized") actually improved generalization in their online setup, consistent with recent deep learning theory.
Key Results
- Offline Accuracy: Directly adding view-attributed conversions to a standard model increased logloss (error) by 3-7%. In contrast, the auto-encoder approach significantly reduced logloss.
- Online Revenue: In a live A/B test involving billions of impressions, the new model achieved a 0.6% CPM lift and a 0.67% total revenue lift.
- Advertiser Impact: There was a 10.1% increase in spend for campaigns using optimized bidding, proving that advertisers were winning more (and better) auctions.
- Zero Latency Penalty: Thanks to the taxonomy-based caching strategy, there was no visible increase in auction latency despite adding a neural network to the pipeline.
Limitations and Future Work
- Feature Depth: The current model uses only high-level taxonomy features to save on computation. The authors suggest that finding ways to include more granular features without hitting latency walls is a major next step.
- Architecture Evolution: While a simple MLP-based auto-encoder worked well, the researchers are looking into Transformers and more advanced self-supervised techniques like Barlow Twins to extract even richer representations from the tabular data.
- Interaction Modeling: Currently, the "code" is added to the model as a linear term. Future versions might explore "triple" inner products to model deeper interactions between the user, the ad, and the self-supervised code.
Summary updated