Funnel-Coupled Sparse Gaussian Processes for Rare Conversions

Ad and e-commerce systems must estimate the probability that a clicked item leads to a purchase. Purchases are extremely rare, often fewer than one in a thousand impressions, and conversion labels exist only for items that were clicked. Current deep models handle the rarity with shared embeddings, extra losses and pseudo-labels for unclicked items. None of them gives a principled answer to a basic operational question: how much should we trust this conversion estimate, here, for this item with three historical purchases? The answer matters three ways. It decides which pseudo-labels on unclicked impressions are safe to train on. It decides whether a cost-per-action bid is backed by evidence. It decides whether a cold-start item is worth exploring. The best current answer, in UKD, measures disagreement between two dropout sub-networks, which is a heuristic and sensitive to the dropout rate.

We propose a small Gaussian process placed on top of a standard conversion model's learned embeddings. A Gaussian process (GP) is a Bayesian model over functions whose predictive variance grows wherever training evidence is thin. Our GP models click and post-click conversion jointly, and it is trained so that rare positives are neither drowned out nor artificially reweighted. Its posterior variance then serves as a calibrated confidence signal. We use that signal to decide which unclicked impressions receive soft conversion labels, and we test it on a benchmark where unclicked-space predictions can be checked against randomly exposed items.

Core Claim

Off-the-shelf scalable GP classification fails on conversion data for two identifiable reasons. Both can be fixed cheaply, and once they are, GP posterior variance beats dropout-based uncertainty as a filter for unclicked-space pseudo-labels and as a calibration tool for sparse items.

The first failure is inducing-point starvation. Sparse GPs summarize the data through $M$ learned anchor points called inducing points. When these are fitted to the global data density, almost all of them land among non-converting impressions, and the sharp, isolated conversion regions get smoothed away. Approximation-Aware Bayesian Optimization documents this oversmoothing of localized structure under a finite inducing budget.

The second failure is likelihood domination. The per-point Bernoulli term in the variational objective of Scalable Variational Gaussian Process Classification is overwhelmed by negatives at below-0.1% positive rates. That paper evaluated only near-balanced data such as MNIST and airline delays. The usual industrial patch is negative downsampling, used for example by DHEN, and it miscalibrates probabilities.

Our fix has three parts. We reserve a stratum of inducing points for converted examples. We couple the click and conversion latent functions through shared inducing variables, so dense click evidence shapes the posterior over rare conversions. We subsample negatives but reweight them so the objective stays an unbiased estimate of the full-data bound.

Let $h_i$ be the backbone embedding of impression $i$, and let $f_c$ and $f_v$ be the click and conversion-given-click latent functions. These two functions are drawn from a two-output GP with a learned 2×2 output covariance, and their inducing variables $u$ sit at locations $Z = Z_+ \cup Z_0$. The set $Z_+$ is initialized at cluster centres of converted embeddings and $Z_0$ at centres of all impressions. The objective is

$$\mathcal{L} = \sum_{i \in S} w_i \, \mathbb{E}_{q(f_c, f_v)}\big[\log p(y_i, y_i z_i \mid f_c(h_i), f_v(h_i))\big] - \mathrm{KL}\big(q(u)\,\|\,p(u)\big).$$

Here $S$ is a minibatch containing all clicked or converted impressions plus non-clicked impressions sampled at rate $r$. The weights are $w_i = 1$ for retained positives and $w_i = 1/r$ for sampled negatives. The likelihood is the entire-space factorization of ESMM, $p(y z = 1) = \sigma(f_c)\sigma(f_v)$, which is non-conjugate. We therefore estimate the expectation by Monte Carlo, following Doubly Stochastic Variational Bayes. The reweighted sum remains an unbiased estimate of the full-data bound because that bound decomposes into independent per-point terms, the property established in Gaussian Processes for Big Data. Downsampling without reweighting does not have this property.

Position Relative to Prior Work

Three papers motivate the proposal. Each shows that the unclicked space carries usable but noisy signal, and none quantifies epistemic uncertainty in a principled way. UKD pseudo-labels unclicked ads with an adversarially adapted teacher and down-weights uncertain labels using dual-dropout discrepancy. IdeFN finds false negatives among unclicked impressions by partial optimal transport to confirmed conversions. That costs $O(N_P N_U)$ and produces hard relabels gated by thresholds. EVI distils a propensity-conditioned teacher through mutual-information objectives in deterministic networks. We keep their shared premise that unclicked impressions should be soft-labeled. We change the uncertainty source to a GP posterior, and we make the labels soft and variance-weighted rather than thresholded.

Two lines of work supply the method. The first is scalable variational GP inference: Gaussian Processes for Big Data for minibatch inducing-point training and SVGPC for its classification extension. The second is the funnel decomposition of ESMM, whose chain rule we move from network outputs into the GP likelihood.

Two papers constrain the design. Approximation-Aware Bayesian Optimization warns that inducing budgets erase isolated peaks, which is why we stratify inducing points. GPnn avoids oversmoothing by conditioning each prediction on nearest neighbours. However, it was evaluated only on low-dimensional regression and depends on nearest-neighbour search, which degrades in high dimensions. We therefore treat it as a competitor rather than our base.

We deliberately leave aside the recurrent streaming GP of OHSVGP. Delayed attribution is real, but it is a separate problem.

Research Program

The first stage is diagnostic. On Ali-CCP we freeze the embeddings of a trained ESMM backbone and fit a standard SVGPC head with uniformly initialized inducing points. We then measure the effect of starvation directly by recording three quantities as a function of the positive rate:

  • how many inducing points end up within a typical lengthscale of any converted example;
  • the posterior mean at held-out converted examples;
  • the calibration error for items with few conversions.

We obtain the positive rate by subsampling conversions to 1%, 0.1% and 0.01% of impressions. This tests the file's open question about whether variational bounds degrade under severe skew, and it does so before we build anything.

The second stage builds the funnel-coupled head with stratified inducing points and the reweighted objective. Training runs on frozen embeddings, with $M$ between 512 and 2048 and embeddings of a few dozen dimensions, so the per-step cost is $O(BM^2 + M^3)$, independent of dataset size. We then add an optional joint fine-tuning variant in which embedding gradients flow through the GP. That variant is known to risk overfitting the kernel, so we report it separately.

The third stage uses the head as a teacher. Each unclicked impression receives the GP posterior predictive conversion probability as a soft target, weighted by a decreasing function of the posterior variance of $f_v$, and a standard tower is trained on clicked plus soft-labeled unclicked data. The student keeps serving latency unchanged. The GP is needed only in offline teaching and, optionally, for calibrating bids on sparse items.

Baselines

The naive reference is ESMM itself: shared embeddings, entire-space factorization, and no uncertainty, with every unclicked impression treated as a conversion negative. All other methods use the same ESMM backbone and features. A method that cannot beat this reference on sparse-item calibration has no case. The ESMM paper reports a 2.56% absolute CVR AUC gain over earlier baselines on the public data, but the file gives no absolute Ali-CCP AUC figure, so we will reproduce one.

The strongest pseudo-labeling competitor is IdeFN. On Ali-CCP it reports an average relative improvement of 1.96% in AUC and an 8.45% relative reduction in LogLoss over the next-best method across MLP, DeepFM and DCNV2 backbones. On KuaiRand-Pure the file reports no number; no reported result was found on Lacuna for IdeFN's KuaiRand-Pure metrics. We run IdeFN from the same backbone with its transport thresholds tuned under the same tuning budget as our variance weighting. Beating it means matching or exceeding its Ali-CCP AUC while improving LogLoss and calibration in the lowest-conversion item bins.

The strongest uncertainty competitor is UKD. It reports roughly 5% uplift in AUC on production data and a 2–3% gain over Monte Carlo dropout, but no isolated Ali-CCP number is given in the file. The fairest comparison holds the teacher's point predictions fixed and swaps only the weighting signal: UKD's dual-dropout discrepancy against our GP variance. Any difference can then be attributed to the uncertainty estimate alone. We also include Monte Carlo dropout and a five-member deep ensemble on the same tower, given matched training compute.

Two ablations isolate the key ingredients. The first is a uniform-inducing, unweighted-likelihood, uncoupled GP head, which is plain SVGPC on embeddings. It isolates the effect of stratification, reweighting and coupling, each also removed one at a time. The second is a linear-kernel version of the same head, which amounts to a Bayesian last layer. It tests whether the nonparametric kernel matters at all.

Why not just…?

Why not just use deep ensembles or UKD's dropout discrepancy? These estimates are cheap and already production-proven: UKD reports a +3.4% online CVR lift. Their weakness is concentrated in the regime that matters. Ensemble members trained on the same few positives can agree confidently in empty regions, because nothing forces their predictions to spread out away from the data. A GP's variance, by construction, returns to the prior away from evidence. Our matched-teacher experiment settles this: if dropout discrepancy ranks unclicked false negatives as well as GP variance does on the randomly exposed KuaiRand split, we drop the claim.

Why not just a Bayesian last layer on the ESMM tower? This is the most serious threat. It is a GP with a linear kernel, cheap and closed-form, and it may capture most of the benefit if the frozen embeddings already make conversions linearly separable. We include it as a direct ablation with identical stratified reweighting. If it matches the RBF head within noise, the honest conclusion is that the stratification and reweighting, not the kernel, do the work. We would then report the method in that simpler form.

Why not just calibrate ESMM post hoc and add a ranking loss? The ranking-loss analysis shows that BCE plus RankNet keeps gradients alive under sparsity and lifts GMV online, and temperature scaling fixes average calibration. Neither, however, gives per-example confidence: a global temperature cannot distinguish a well-supported item from one with three purchases. Pseudo-label filtering needs exactly that distinction. We include Combined-Pair plus temperature scaling as a baseline, and we evaluate calibration stratified by item conversion count, where a global correction should fail.

Why not just use local nearest-neighbour GPs? GPnn keeps local peaks intact and scales to 1.6 million points in about 30 seconds on regression. It has not been formulated for binary likelihoods, and its neighbour search degrades with embedding dimension. We run a GPnn-style variant that treats the click-conversion label as a regression target on the same embeddings. If it matches the coupled head, locality alone suffices, and that result would itself be informative.

Proposed Experiments

The datasets are Ali-CCP and KuaiRand-Pure. Ali-CCP supplies click and conversion labels with genuine extreme sparsity. KuaiRand-Pure is valuable because its randomly inserted videos give missing-at-random exposures. On those exposures we can evaluate predictions for items the logging policy would never have shown, which directly tests whether unclicked-space soft labels are right. KuaiRand has no purchases, so we designate a rare post-click signal from its twelve feedback types as the conversion target, fix that choice before seeing results, and report its base rate.

The metrics are CVR AUC and CTCVR AUC, LogLoss, and expected calibration error both overall and within item bins grouped by training-conversion count. For the pseudo-labeling claim we also report how well uncertainty ranks errors: the precision of the retained soft labels at a fixed retention rate on the random-exposure split.

The first decisive check is the starvation diagnostic. If plain SVGPC is already well calibrated at 0.01% positives, the stratification motivation disappears. The second is the matched-teacher swap of GP variance against dropout discrepancy on KuaiRand's random exposures. The third is the linear-kernel ablation, which says whether the nonparametric component earns its place. The fourth is the end-to-end comparison with IdeFN on Ali-CCP, which shows whether calibrated uncertainty translates into better student models without IdeFN's quadratic transport step.

We will report every comparison with seeds and matched tuning budgets. We will report negative outcomes on the second and third checks as findings rather than bury them.