When crowd workers label images taken in rain, fog, snow or low light, they tend to make the same mistakes on the same images. A fogged pedestrian gets called a pole by most of the crowd, and a streaked, veiled truck gets called a bus. Standard ways of combining labels, from majority vote to the Dawid-Skene model (which estimates each worker's confusion matrix and assumes workers err independently once the true label is fixed), read that shared mistake as strong agreement. They then hand confident wrong labels to whatever detector is trained next. Existing correlation-aware methods can represent shared confusion, but they cannot say where it comes from. A class boundary that is genuinely ambiguous and a physical corruption that could be removed look the same to them, because both are visible only as label agreement patterns.
We propose to measure the dependence directly. On a small calibration subset, the same scene is shown to annotators at a graded ladder of compound degradation severities, rendered from a clean original. Shared error that is present on the clean rung is semantic ambiguity. Shared error that grows with the degradation mixture is degradation-induced. We fit a label-aggregation model in which the strength of shared confusion is an explicit function of a learned description of the degradation mixture, taken from an all-in-one restoration encoder. That function then transfers to real adverse-weather images that have no clean counterpart. The result is an aggregator that knows when agreement on a degraded image should not be trusted. It also gives a stopping rule for adaptive labeling that stops treating correlated workers as independent evidence.
Core Claim
The claim is that shared annotator error on compound-degraded images is largely predictable from the degradation itself, and that controlled variation of degradation on the same content makes this dependence identifiable. Label co-occurrence alone cannot identify it. If the claim holds, conditioning shared confusion on a degradation descriptor, fitted with ladder data, should cut the rate of confident-but-wrong consensus labels well below what EBCC or CoNAL achieve on the same labels. It should also make stopping rules for adaptive labeling spend more annotations on exactly the images where the crowd is jointly fooled.
What Exists and What Is Missing
Three lines of work already relax the independence assumption, and the proposal builds most directly on them. Exploiting Worker Correlation for Label Aggregation in Crowdsourcing (EBCC) splits each true class into latent subtypes. Its own example is "clear dog" versus "blurry dog". Workers correlate because they share a vulnerability to a subtype, and the model stays linear in the number of workers. Learning from Crowds by Modeling Common Confusions (CoNAL) separates a global confusion matrix from per-worker ones and learns an instance-specific weight saying how much of each label comes from shared confusion. It reports that 61% of LabelMe annotators made the same "street" to "inside city" error. Active Learning from Crowds makes worker expertise a function of input features, but it keeps errors independent given those features. These papers motivate the proposal: they show shared confusion is real and tied to image quality. They also supply the method, since our model is a subtype mixture in the style of EBCC. None of them can separate reducible from irreducible shared error. EBCC's subtypes are inferred from label patterns only. CoNAL's weight is fitted to labels, so it absorbs semantic and physical ambiguity alike. STABLEVAL explicitly treats disagreement as legitimate item ambiguity, which is exactly the reading we argue is wrong for degraded images.
Two papers show what goes wrong in vision pipelines that ignore this. Lean Crowdsourcing combines humans and a vision model with a risk-based stopping rule. It observes that under visual ambiguity both converge on the same wrong label, so collection stops early with false confidence. Bayesian Detector Combination (BDC) aggregates crowd boxes and classes for detection and states that it assumes independent raters. These define the failure we target and the detection setting we extend.
On the degradation side, we reuse existing encoders rather than building a new restorer. AirNet learns a degradation representation by contrasting patches from the same versus different degraded images, without degradation labels. The component classifier in Blind Image Decomposition identifies which of rain, raindrop, snow and haze are present with over 90% accuracy. Both representations have only ever been used to steer restoration. We use them as the conditioning signal for annotator confusion. Image-Adaptive YOLO constrains the design. Enhancement tuned for detector loss produces artifacts that look wrong to humans, so "restore, then ask humans" could create new shared errors rather than remove old ones.
The Model
For item $i$ with true class $z_i$ and degradation descriptor $\phi(d_i)$, the descriptor is produced by a frozen AirNet encoder concatenated with BIDeN's component probabilities. We model the joint labels of the $R$ workers as
$$ P(y_{i1},\dots,y_{iR}\mid z_i,d_i)=\sum_{c}\pi(c\mid z_i,d_i)\prod_{r=1}^{R}\theta_r(y_{ir}\mid c),\qquad \pi(c\mid z,d)=\mathrm{softmax}_c\big(a_{zc}+b_{zc}^{\top}\phi(d)\big), $$
Here $c$ is a shared "perceived class" that all workers see for that item, and $\theta_r$ is worker $r$'s individual confusion given what they perceive. Correlation between workers comes entirely from randomness in $c$. The term $a_{zc}$ captures confusion present on clean images, which is semantic ambiguity. The term $b_{zc}^{\top}\phi(d)$ captures confusion that grows with the degradation mixture. Without ladder data, $a$ and $b$ trade off against each other and against per-item difficulty. With ladder data, the same $z$ is observed at several known severities, including $\phi(d)=\phi(\text{clean})$, so the two terms are pinned separately. The ladder plays the role of a controlled intervention on $d$ with the content held fixed.
We fit the model by variational EM, initialized from majority vote. EBCC reports that it is sensitive to initialization and can collapse a class into another. The ladder subset gets anchor labels from the clean rung, and non-ladder items are fitted jointly. The posterior over $z$ feeds two uses: soft training labels, and a stopping rule that stops querying an item only when its posterior risk under the correlated model falls below a threshold. We extend the classification head of BDC with the same $\pi$ so that box aggregation also gets degradation-conditioned shared error.
Research Program
The data construction uses benchmarks with gold labels and controllable degradation. The main testbed is BID Task II from Blind Image Decomposition. It composites rain streaks, raindrops, snow and haze onto Cityscapes, so the clean image and Cityscapes gold labels are both available, and each component's presence and intensity can be set per image. VOC_Foggy from Image-Adaptive YOLO gives a second synthetic set with VOC gold. WeatherBench provides real, physically generated rain, haze and snow paired with clean static scenes. It lacks object labels, so we annotate a subset with experts on the clean images to serve as gold. It is the key test that ladders built from physical rather than composited weather behave the same way.
The labeling task is object-crop classification plus presence, which keeps human cost bounded; boxes are a second stage. Each item gets five to seven crowd labels. About 10% of items are ladder items, shown at four severities, with different workers per rung so that no one sees the clean version first. We run the whole program twice. The first run uses a pool of pretrained detectors of different architectures as machine annotators, which is cheap and lets us iterate. The second uses humans, because shared human perceptual failure is the real target, and because Lean Crowdsourcing shows that human and machine errors coincide under ambiguity.
Deployment tests use RTTS (real fog) and ExDark (real low light), where no clean version exists. The descriptor must transfer from composited or physical ladders to real weather.
Baselines
The naive reference is majority vote over the same five to seven labels per item. As a secondary naive reference, we train on each image's labels without aggregation. Both are run on identical label sets. The published point of comparison is EBCC's synthetic correlated setting, where majority vote reached 72.0% against EBCC's 85.9% (Exploiting Worker Correlation). No reported result exists on Lacuna for any aggregator on BID Task II, VOC_Foggy or WeatherBench crowd labels, so all numbers on our data will be new.
Dawid-Skene and iBCC are the independent-error Bayesian references. They show how much of the gap comes from modeling correlation at all.
EBCC is the strongest aggregation competitor for classification labels. Its subtypes can in principle learn "degraded pedestrian" from label patterns alone, which is exactly our claim restated without degradation grounding. It reports 84.5% mean accuracy across 17 real datasets versus 83.4% for iBCC-MF. We run its empirically initialized ten-subtype variant with the same number of mixture components as our $c$, on the same labels, with the same restart budget.
CoNAL with the degradation descriptor fed into its instance-weight network is the most dangerous competitor, because it uses the same image-side information we use. It reports 87.12% on LabelMe. Both it and our model get the same frozen encoder, the same backbone for the downstream classifier, and the same hyperparameter search budget. The only difference is whether ladder data informs the shared-confusion term.
For detection, the reference is BDC with a YOLOv7 backbone, which reports 19.2 AP against 13.9 AP for majority vote on VinDr-CXR (Bayesian Detector Combination). We compare BDC against BDC with our degradation-conditioned class head, on crowd boxes over the BID Task II images.
The key-ingredient ablations are three. Our model without ladder items tests whether the intervention, not the descriptor, does the work. Our model with the descriptor replaced by a single scalar severity tests whether knowing the mixture composition matters. Our model with a shuffled descriptor controls for extra parameters.
To beat the baselines, our model must have a lower false-consensus rate at matched accuracy than EBCC and than CoNAL-with-descriptor, with confidence intervals from item-level bootstrap. It must also not be worse on clean items. A gain that appears only in aggregate accuracy but not in false consensus would not support the claim.
Why not just…?
Why not just restore the images with AirNet or M2Restore and then aggregate normally? This is a real alternative, and we run it: annotators label restored images, followed by majority vote and EBCC. There are reasons to expect it to fall short. Detector-tuned enhancement creates artifacts humans find unnatural (Image-Adaptive YOLO). Generative restorers can invent detail, and an invented detail seen by every worker is a new shared error. The ladder design measures this directly: we compare the correlation of worker errors on restored images against the clean and degraded rungs. If restoration brings shared error down to clean-rung levels, restoration alone is sufficient for synthetic mixtures and our contribution narrows to real weather, where restorers trained on synthetic data transfer poorly (WeatherBench reports AdaIR at 18.87 dB with real training versus 18.28 dB with synthetic). Our model can also take restored images as input, so the two are complementary rather than exclusive.
Why not just collect more labels per image? Averaging only cancels independent errors. EBCC's synthetic correlated setting already shows majority vote well behind a correlation-aware model, and Lean Crowdsourcing shows that high redundancy is wasteful where errors are independent. We test this directly by plotting majority-vote accuracy on heavy-degradation rungs as redundancy grows from 3 to 15 labels. The hypothesis predicts a plateau below the clean-rung accuracy. If the curve instead keeps climbing, degradation-induced errors are closer to independent than we think and the proposal loses its motivation.
Why not just give CoNAL or EBCC degradation features and skip the ladder? This is the strongest threat, because feature-conditioned shared confusion is only a small step from CoNAL. Our argument is identifiability. From labels alone, a degraded item where 70% of workers agree on the wrong class cannot be told apart from a genuinely ambiguous item with the same agreement pattern. The no-ladder ablation and CoNAL-with-descriptor settle it. If they match our false-consensus rate, the ladder is unnecessary, and the honest result is "condition CoNAL on a restoration encoder". That is still a useful finding, but a smaller one.
Why not just drop heavily degraded images from the training set? Those images are the deployment distribution for adverse-weather perception. Dropping them removes exactly what RTTS and ExDark test. We include a filtered-training baseline for the downstream detector to measure the cost.
Decisive Checks
The first check comes before any modeling. On the machine-annotator pool and then on a pilot human pool, we measure pairwise error correlation among workers as a function of degradation severity and composition, conditioned on the true class. If correlation does not rise with compound degradation above its clean-rung level, the premise is false and we stop.
The second check is held-out mixtures. We fit on ladders built from rain streaks plus haze and test false-consensus rate on rain plus snow plus haze and on raindrops, which tests whether the descriptor, rather than memorized mixture identities, carries the dependence.
The third check is the stopping rule. Following the Lean Crowdsourcing protocol, we compare labels spent per item and final error when the risk estimate uses our correlated posterior versus the independent one. The prediction is that our rule spends more labels on heavily degraded items and fewer on clean ones, at lower final error.
The fourth check is real transfer. We train YOLOv3 on labels aggregated each way from crowd annotations of VOC_Foggy and BID Task II images, then evaluate on RTTS and ExDark. The reference points are Image-Adaptive YOLO's reported 28.82% mAP for YOLOv3 and 37.08% for IA-YOLO on RTTS, trained on single gold labels. That training regime differs from ours, so these values mark the range rather than a like-for-like comparison. We will report whether better aggregation closes part of the gap between crowd-label training and gold-label training.
References
- Exploiting Worker Correlation for Label Aggregation in Crowdsourcing
- Learning from Crowds by Modeling Common Confusions
- Active Learning from Crowds
- STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems
- Lean Crowdsourcing: Combining Humans and Machines in an Online System
- Bayesian Detector Combination for Object Detection with Crowdsourced Annotations
- All-In-One Image Restoration for Unknown Corruption
- Blind Image Decomposition
- Image-Adaptive YOLO for Object Detection in Adverse Weather Conditions
- WeatherBench: A Real-World Benchmark Dataset for All-in-One Adverse Weather Image Restoration