When each image or sentence gets only one to three crowd labels, a classifier trained on the data can act as an extra, tireless voter. A recent method, Domain-Aware Voting, does exactly this. It turns a classifier's predicted class probabilities into a prior, a set of "pseudo-votes" added to the real crowd votes. One global knob controls how many pseudo-votes the classifier gets. The weakness is that human crowd errors are not random. They cluster on ambiguous items, and those are the same items where a classifier trained on crowd labels has memorized the crowd's mistakes. On those items the classifier's vote is confidently wrong, and with only a few human labels nothing outvotes it.
We propose to learn, from the crowd labels themselves, how many pseudo-votes the classifier deserves on each kind of instance. Items that received several labels act as a free audit. On them we can measure how often the classifier's confident predictions are contradicted by humans. That measured disagreement then sets the classifier's weight on the many items that received only one label. Two further ingredients make this work. The classifier is cross-fitted, meaning it never scores an item whose own labels it was trained on. It is also refreshed as labels arrive. The result replaces a fixed weight with a weight that falls off according to observed contradiction, and it requires no ground-truth validation set.
Core Claim
The prior's influence should shrink according to how much the data contradicts it, not according to how many labels have arrived. Choosing that influence by marginal likelihood on multiply-labeled items, using a classifier that has not seen those labels, fixes three things at once. It recovers most of the accuracy lost by fixed-weight classifier priors on instance-dependent human noise. It removes the unsolved problem of tuning without gold labels. It does both with almost no computational overhead.
Where This Sits
The direct starting point is On the use of the descriptive variable for enhancing the aggregation of crowdsourced labels. That paper builds Domain-Aware Voting, in which a classifier's output becomes a conjugate Dirichlet prior with strength $\alpha_0$. The prior is overridden only as an item gathers more labels. The authors name tuning $\alpha_0$ without ground truth as an open problem. Their classifier is trained once, on the same noisy labels it then votes on, and nothing checks whether its beliefs are wrong. We keep their conjugate model and change how its strength is set.
The motivating evidence against fixed-weight priors comes from Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. That work shows on CIFAR-10N and CIFAR-100N that real crowd noise depends on the instance, and that networks memorize it faster than synthetic noise. This is exactly the condition under which a classifier prior is wrong in a structured way, rather than wrong at random.
The design idea is borrowed from Bayesian optimization, where expert priors are faded out over the course of a search. πBO raises the prior to a power $\beta/n$ that shrinks with iteration count. DynMeanBO shrinks a prior mean exponentially with iteration count. Both use time-based schedules, like Domain-Aware Voting's count-based fading. Two other methods react to evidence instead. PriorBand shifts sampling away from the prior when data contradicts it. DynaBO rejects priors that the surrogate model predicts will underperform. PriorBand also reports that πBO's schedule fades too slowly under bad priors. Our proposal carries this "react to evidence" principle into label aggregation. DynaBO's own open problem warns that a safeguard is only as good as the model judging the prior. That warning is why we judge the prior against held-out human labels and not against a model fit to them.
Two results constrain the design. The first is Robustness of Bayesian Pool-Based Active Learning Against Prior Misspecification. It proves that degradation stays graceful only when the utility is Lipschitz in the prior, meaning small errors in the prior cause only small changes in the utility. It proposes mixing the prior with a uniform distribution. The second is the "grain of truth" condition behind Thompson Sampling for Complex Online Problems: recovery is impossible if the prior gives the truth zero probability. Both lead us to mix every classifier prior with a uniform floor. That way no class is ever ruled out.
The Mechanism
For item $i$, let $p_i$ be the cross-fitted classifier's class distribution mixed with a uniform floor. Let $n_i$ be the vector of crowd label counts for that item, weighted by each worker's estimated reliability as in Dawid–Skene-style aggregation. Let $N_i$ be its total. The aggregated soft label is the Dirichlet posterior mean:
$$\hat{q}_i = \frac{s(x_i)\,p_i + n_i}{s(x_i) + N_i},$$
where $s(x_i)$ is the prior strength, the number of pseudo-votes. In Domain-Aware Voting, $s \equiv \alpha_0$ is a single constant. We instead let $s$ be a small function of the classifier's state. Its inputs are the predicted confidence, the predicted entropy, and the disagreement among the cross-fitting folds. We fit $s$ by maximizing the Dirichlet-multinomial marginal likelihood of the labels on items with at least two labels:
$$\max_{s}\ \sum_{i:\,N_i \ge 2} \log \mathrm{DirMult}\big(n_i \mid s(x_i)\,p_i\big).$$
This objective rewards a large $s$ where the classifier agrees with the humans and pushes $s$ toward zero where confident predictions are routinely contradicted. Most real crowd sets have a few multi-labeled items mixed with many singly labeled ones. The function $s$ learned on the former is then applied to the latter.
Cross-fitting closes the loop that would otherwise let the classifier "confirm" labels it memorized. Labels are collected in rounds. After each round we retrain the K cross-fitting classifiers and refit $s$. We also optionally spend the next label budget on items where the posterior's argmax would most likely flip. Because that flip probability depends on the learned $s$, label budget naturally flows toward the conflicted items.
Research Program
We first build the estimator and measure, on CIFAR-10N, how well the learned $s(x)$ tracks the true per-bin rate at which the classifier is wrong. Clean CIFAR-10 labels are available for this check. Next we run sequential label-collection simulations by revealing crowd labels one at a time. We use the 3 labels per image in CIFAR-10N and the roughly 50 per image in CIFAR-10H, which gives access to the regime where To Aggregate or Not? Learning with Separate Noisy Labels finds that aggregation stops being preferable. Finally we test transfer to LabelMe and the real crowdsourced sets used by Domain-Aware Voting, and to the UCI datasets with simulated noise that depends on the instance.
We evaluate three things. The first is the accuracy of the aggregated labels against ground truth, reported separately for items with 1, 2, and 3 or more labels. The second is the downstream test accuracy of a network trained on the aggregated soft labels. The third, on CIFAR-10H, is the divergence between our soft labels and the dense human label distribution.
Baselines
The naive reference is majority voting. On LabelMe, Deep Learning from Crowds reports 76.74% accuracy for majority voting, against 82.67% for EM-based joint learning and 83.15% for its Crowd Layer. The Crowd Layer appends a learned confusion matrix per worker to a network trained end to end. The 83.15% figure is the strongest reported number in the file on a benchmark we share, and beating it on LabelMe means exceeding 83.15% test accuracy under the same backbone and training schedule.
The most direct competitor is Domain-Aware Voting with a fixed $\alpha_0$, run in two versions. One uses the authors' default. The other has $\alpha_0$ tuned on a small gold-labeled subset, which gives it information our method never sees. No reported result found on Lacuna for Domain-Aware Voting accuracy on any benchmark, so we will reproduce it using their Random Forest-style classifier and our classifier, keeping whichever is stronger.
We also run classical latent-variable aggregators. These are Dawid–Skene and GLAD from Whose Vote Should Count More, which models per-item difficulty and per-worker expertise. They use no classifier prior. We also include training on separate, unaggregated labels following Wei et al. Their reported result is that on CIFAR-10N, with 3 labels per image, separation consistently beats aggregation. For context, standard cross-entropy on CIFAR-10N "Worst" reaches about 77% according to the CIFAR-10N study.
All methods share the same backbone, the same number of training epochs, and an equal hyperparameter search budget. For our method, that budget covers only the uniform floor and the capacity of $s$.
Three ablations isolate the key ingredients. The first keeps cross-fitting but uses a single global $s$ chosen by marginal likelihood. This tests whether making $s$ depend on the instance matters. The second keeps the learned $s(x)$ but drops cross-fitting. This tests the confirmation-bias claim. The third replaces evidence-driven $s$ with a count-based schedule in the style of DynMeanBO.
Why not just…?
Why not just tune $\alpha_0$ on a small gold set? This is the most serious threat, and we run it as a baseline. Our predicted advantage comes from two sources. First, a single scalar cannot be large on items where the classifier is reliable and small on ambiguous ones. Second, gold sets are often unavailable, which is the very gap the Domain-Aware Voting authors name. If gold-tuned fixed $\alpha_0$ matches the learned $s(x)$, the honest conclusion is that our contribution reduces to label-free tuning of $\alpha_0$. That is still useful, but narrower, and the global-$s$ ablation will show whether this is the case.
Why not just drop the classifier prior and use Dawid–Skene or the Crowd Layer? These models capture how individual workers err but do not use the input features of single-labeled items. Domain-Aware Voting reports that it helps most at 1–3 labels per item, where per-worker confusion matrices are poorly estimated. If our method fails to beat the Crowd Layer's 83.15% on LabelMe, the prior is not earning its place there.
Why not just use a noise-robust classifier, or temperature-scale its confidence, as the prior? The CIFAR-10N study reports 1–5% drops for DivideMix and ELR+ when moving from synthetic to human noise. Their errors therefore fall on the same ambiguous items, and calibration rescales confidence globally without locating those items. We will test this directly by substituting an ELR+-trained classifier and a temperature-scaled classifier into fixed-$\alpha_0$ Domain-Aware Voting.
Why not just collect more labels? On CIFAR-10H we can measure exactly how many extra labels per item fixed-weight Domain-Aware Voting needs to match our method. This turns the advantage into an annotation-cost saving rather than an accuracy claim.
Decisive Checks
The idea stands or falls on a few checks. On CIFAR-10N "Worst" (about 40% noise), consider the items where the classifier is confident but wrong according to clean labels. The learned $s$ on those items must be significantly lower than on items where it is confident and right. On items with a single label, aggregated accuracy must beat gold-tuned fixed-$\alpha_0$ Domain-Aware Voting. Removing cross-fitting must measurably reduce that gain. If the first check fails, the marginal likelihood is not detecting misspecification and the proposal is wrong. If only the third fails, the weight that reacts to evidence is doing the work and the confirmation-bias story is not.
References
- On the use of the descriptive variable for enhancing the aggregation of crowdsourced labels
- Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
- $\pi$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
- Incorporating Expert Priors into Bayesian Optimization via Dynamic Mean Decay
- PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
- Dynamic Priors in Bayesian Optimization for Hyperparameter Optimization
- Robustness of Bayesian Pool-Based Active Learning Against Prior Misspecification
- Thompson Sampling for Complex Online Problems
- To Aggregate or Not? Learning with Separate Noisy Labels
- Deep Learning from Crowds
- Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise