Detection Limits for Sparse Utility Labels in Click-Dominated Rankers

When a lab test checks a single sample for many pathogens at once, a common pathogen can use up the shared reagents and push a rare one below what the instrument can detect. Search and recommendation models face a similar problem. A ranker trained on millions of clicks and a few thousand expert relevance judgments puts both signals through one shared network. Clicks are the "high-copy target" and swamp the shared capacity. Expert relevance is the "low-copy target": it ends up underfit, or ignored entirely. The symptom is well documented. On Baidu's 1.2-billion-session search log, click-debiasing methods improve click prediction but not expert-judged ranking quality, and plain BM25 often wins (Unbiased Learning to Rank Meets Reality). In health search, click-derived and expert system rankings agree only at Kendall's Tau 0.238 to 0.428 (TripJudge).

We propose to borrow the diagnostic lab's central measurement, the limit of detection, and apply it to rankers. The question is how many expert labels a jointly trained model needs before it can reliably "detect" expert relevance, and how that number shifts as click data dominates and as more objectives are packed into the same model. We then test one specific fix. Each objective is written as a short natural-language description ("the document fully answers the medical question", "a user would click this result") and given to the model as part of its input, rather than as an anonymous output head. In other words, each objective becomes a description the model reads. The goal is to find out whether this lowers the detection limit for the sparse expert criterion compared with ordinary multi-task training, gradient-balancing methods, and simply training on expert labels alone.

Core Claim

The misalignment between engagement and utility is partly a capacity-sharing failure, and that part can be measured. The measurement is a detection-limit curve. For a target expert-relevance score $s^\*$ on held-out queries, define

$$n^\*(r, K) = \min \{ n : \text{nDCG@10}_{\text{expert}}(n, r, K) \ge s^\* \},$$

where $n$ is the number of expert-labeled training queries, $r$ is the ratio of click-labeled to expert-labeled training pairs, and $K$ is the number of objectives trained jointly. Sharing a backbone has two effects that pull against each other. Transfer from clicks should lower $n^\*$. Starvation by clicks should raise it as $r$ and $K$ grow, possibly sharply, by analogy with the capacity threshold that Multi-Node Graphs proves for multiplex PCR panels and the accelerating accuracy loss DataMUX reports when many inputs share one Transformer pass.

Our hypothesis is that conditioning on semantic objective descriptions keeps the transfer benefit while delaying starvation. The reasoning is that the description works as an input-side filter that tells the shared encoder which evidence to use. This is the "hard attention" effect that Description Based Text Classification credits for cutting BeerAdvocate multi-aspect error from 27.8% to 15.6%. The claim is new in two respects. Nobody has measured detection limits for sparse utility labels under click dominance. Nobody has tested whether describing objectives in language moves those limits. Both are falsifiable.

Research Program

The model is a single cross-encoder, meaning a Transformer that reads the query and document together and outputs a score. It takes the input [criterion description] [SEP] query [SEP] document. The criteria come from the supervision that is actually available. One is "a user clicked this result", supervised by TripClick logs (TripClick) or Baidu-ULTR clicks. The others come from the graded TripJudge scale, whose levels (Wrong, Topic, Partial, Perfect) map onto descriptions such as "the document is on the query's topic" and "the document fully answers the query". To grow $K$ for the multiplexing sweep, we add further engagement-style criteria derivable from logs, such as click position and session continuation, so that the dense-signal side gets more crowded.

Three questions structure the work. The first is whether starvation exists and what shape it takes. We sweep $r$ over several orders of magnitude and $K$ from 2 upward, and trace $n^\*$ for a standard multi-head model. The second is whether descriptions move the curve, and why. The key ablation replaces each description with an arbitrary index token ("criterion 3") in the same input slot. This separates the semantic content of the description from the mere fact of input-side conditioning. A second ablation shuffles descriptions across criteria. The third is mechanistic. We track per-criterion gradient norms through training, in the spirit of the imbalance diagnosis in IMGrad, to see whether a rising $n^\*$ coincides with the expert criterion's gradient share collapsing early. This is the neural analogue of early-cycle reagent exhaustion modeled by DYNAMIC.

We use the PCR literature only as a source of measurements and hypotheses (detection limits, threshold behavior, early-depletion dynamics). We do not import its kinetic equations, because nothing in the file shows those equations transfer.

Before touching noisy logs, we use BeerAdvocate as a controlled testbed. Every aspect there has dense labels, so we can artificially subsample one aspect (say aroma) to make it "low-copy" while keeping the others dense. This gives clean detection-limit curves where the ground truth is never in doubt, and it lets us check that our description model reproduces the reported Chai et al. result before we extend it.

Baselines

The naive reference point is a cross-encoder trained on clicks only and evaluated on expert labels. This is the model that both Hager et al. and Althammer et al. show failing to transfer. Beating it is necessary but uninteresting.

The strongest single competitor on expert relevance is BM25. TripJudge reports that it outperforms SciBERT on human judgments, and Unbiased Learning to Rank Meets Reality reports that it outperforms many click-trained neural rankers on expert DCG@10. Neither summary in the file gives the numeric score: no reported result found on Lacuna for BM25 nDCG@10 on TripJudge or DCG@10 on Baidu-ULTR. We therefore rerun BM25 ourselves on identical query splits. Any neural method that fails to exceed BM25 on expert nDCG@10 at a given $n$ is treated as having not detected the expert criterion at that $n$. In practice we may set $s^\*$ relative to our measured BM25 score.

The strongest learned competitors are three. The first is a multi-head multi-task cross-encoder with tuned fixed loss weights, mirroring the co-engagement plus semantic-relevance design of Balancing Semantic Relevance and Engagement. That paper reports semantic relevance rising from 35.1% to 46.9% on a proprietary video dataset, which we cannot reuse. The second is the same model trained with IMGrad, the strongest published gradient-imbalance method in the file (NYUv2 $\Delta m\%$ of −4.57% versus −4.04% for Nash-MTL, a vision benchmark, so it is not comparable to our numbers). The third is sequential training: click pretraining followed by fine-tuning on expert labels only.

On BeerAdvocate, the strongest reported number is description-based BERT at 15.6% error (Description Based Text Classification). Our full-label run must match this before the subsampled runs mean anything.

Fairness is enforced as follows. Every method uses the same backbone, the same training-step budget, the same click and expert pools at each $(n, r)$, and an equal hyperparameter search budget (the same number of trials, including loss weights for the multi-task baselines). Every method is evaluated by 5-fold cross-validation over TripJudge's 1,136 queries and over held-out partitions of Baidu-ULTR's 6,985 expert-annotated queries, with fold assignments fixed across methods.

"Beating" a baseline means a lower $n^\*$ at matched $(r, K)$. Gains in single-point nDCG must be reported with confidence intervals across folds.

Why not just…?

Why not just train on the expert labels alone? Singleplexing is the honest null. If the sparse criterion never shares capacity, it cannot be starved. The answer depends on whether transfer from clicks lowers $n^\*$ below the expert-only curve. Nothing in the file settles this, and the click-to-expert disagreement in TripJudge suggests transfer may be weak. The expert-only curve is therefore plotted in every experiment. If it matches or beats every joint model, we report that, and the practical conclusion becomes "don't multiplex utility labels with clicks".

Why not just pretrain on clicks and then fine-tune on expert labels? This is the most serious threat. It gets transfer without simultaneous competition. If sequential training reaches the same $n^\*$ as description-conditioned joint training, then the starvation story is real but the cure is simpler than ours, and we would recommend sequential training instead. We expect joint training to matter only when the deployed model must keep serving the click criterion too, because fine-tuning may erode it. So we also measure click NLL after fine-tuning, and judge both curves together.

Why not just fix the imbalance with gradient balancing? IMGrad directly targets dominant tasks starving weak ones and works on vision and robotics benchmarks. If IMGrad applied to the multi-head model closes the gap, the semantic descriptions are unnecessary. The index-token ablation combined with IMGrad tells these explanations apart. Descriptions earn their place only if they lower $n^\*$ beyond what IMGrad achieves. They should also lower $n^\*$ at $n = 0$, where a description model can make zero-shot judgments and an untrained head cannot.

Why not just use BM25? On expert labels it is already competitive, per both the Baidu and TripJudge findings. It is the floor every learned method must clear. It is also a feature we will feed to all models, so that our gains cannot be explained by lexical signal the neural models failed to pick up.

Proposed Experiments

The first decisive check is on BeerAdvocate. We subsample one aspect to between 1% and 100% of its labels while the other aspects stay dense, and compare detection-limit curves for multi-head BERT, index-conditioned BERT, and description-conditioned BERT. This answers whether semantic descriptions lower $n^\*$ under controlled dominance.

The second check sweeps $r$ and $K$ on TripClick/TripJudge and Baidu-ULTR to test whether expert $n^\*$ rises gradually or past a threshold. A threshold would echo the capacity transition in Multi-Node Graphs. A gradual rise would be a negative result for the analogy, and we would report it as such.

The third check compares description conditioning against IMGrad, sequential fine-tuning, and expert-only training at matched budgets, alongside the gradient-share traces.

The proposal succeeds if description conditioning gives a lower expert $n^\*$ than every learned baseline in at least the high-$r$ regime, beats our measured BM25 score, and keeps click NLL within the multi-head model's range. It fails cleanly if expert-only or sequential training matches it. Either result tells practitioners how to spend scarce expert labels.