Anchoring Follows Confusion: How Machine Label Suggestions Fail Where Classes Look Alike

When datasets are labeled with help from a model, the model usually proposes a label and a person confirms or corrects it. Everyone knows people sometimes rubber-stamp wrong suggestions. This proposal argues that in fine-grained recognition this is a structured problem, not scattered noise. Fine-grained recognition means telling apart sibling categories that look nearly identical, such as two flycatcher species or two trims of the same car. Annotators should be most easily talked into a wrong label exactly when the wrong class looks like the right one, and those are the same class pairs the model itself confuses. If so, model-assisted fine-grained datasets carry label errors laid along the model's own confusion pattern. Such errors would be hard to detect. They would make models from the same family look better than they are. And they would train the next model to repeat the confusion.

We propose a controlled annotation study to test this claim, and a small interface change to counter it. In the study, the same image is shown with a randomly assigned suggestion: correct, wrong but visually close, or wrong and visually distant. This separates the pull of the suggestion from the difficulty of the image. The interface change replaces "here is the model's label, confirm it" with "which of these two look-alike classes is it?". The pair shown is the model's pick and its most confusable sibling, in random order and with no marker of which one the model preferred. The goal is to keep the narrowing-down that makes assistance useful while removing the anchor. A simulation stage then asks what this kind of noise does to downstream models and benchmarks.

Core Claim

The probability that an annotator accepts a wrong suggestion rises steeply with the visual similarity between the suggested class and the true class. For near siblings, the assisted error rate exceeds the from-scratch error rate on the same images. The resulting human errors line up with the suggesting model's confusion matrix more closely than unassisted human errors do. Written as the model we will fit,

$$P(\text{accept} \mid \text{wrong suggestion}) = \sigma(a + b\,s + c\,e + d\,s\,e),$$

where $\sigma$ is the logistic function, $s$ is the similarity between suggested and true class (measured on held-out embeddings and on the model's confusion rate for that pair), and $e$ is annotator expertise. The claim is that $b$ is large and positive, and that expertise weakens but does not remove it ($d<0$, while $b+d>0$ at the highest expertise tier).

Why this is not already known

The anchoring evidence so far comes from settings where class similarity was never varied.

In syntax, Anchoring and Agreement in Syntactic Annotations showed that parsers score higher on data their own outputs seeded. Error rates drop by 34–49% on self-corrected gold, and the parser gains 3.88% labeled attachment score. That work motivates our benchmark-inflation test, but it says nothing about which errors survive correction.

In pathology, two papers bracket the question. Are pathologist-defined labels reproducible? used detector proposals to recover missed mitoses and noted confirmation bias without quantifying it. Information mismatch in PHH3-assisted mitosis annotation found that an assistive stain raised agreement between pathologists from F1 0.53 to 0.74 while lowering downstream detector precision. In that study, 172 of 308 newly added figures were unrecognizable in the target stain. Both pathology studies are essentially binary detection tasks with a non-model assistant or an unmeasured one.

The closest study in design, Does AI Assistance Preserve or Collapse Disagreement?, is a real counterweight. On coarse video categories, AI help tightened boundary timing, yet semantic label entropy stayed flat (1.18 vs 1.17) and annotators changed about half the suggested labels. We take that finding seriously. Our hypothesis is that suggestions act as a scaffold when classes are distinguishable and as an anchor when they are not. The similarity manipulation is the variable that should reconcile the two results.

Finally, Visual Recognition with Humans in the Loop built the reference human-machine system for 200 bird species, raising accuracy from about 19% to about 66%. It models the person as an independent noisy sensor. That is exactly the assumption we test.

Research Program

Audit set. We will build a from-scratch gold audit set from confusable pairs. From Caltech-UCSD Birds-200-2011 we select about 60 sibling pairs, spanning high to low similarity, as ranked by the confusion matrices of two unrelated backbones (a CLIP-style model and a DINOv2-style model). We add about 30 pairs from Stanford Cars and FGVC-Aircraft, for roughly 1,500 images. Three domain experts label every image independently, from scratch and without suggestions. Images without consensus are kept as a separate "indeterminate" stratum rather than discarded. This echoes the pathology finding that some cases carry no visual evidence, and it lets us check whether assistance manufactures confident labels for images that have none.

Controlled suggestion study. About 60 annotators take part, split between lay crowd workers and experienced hobbyist birders or car enthusiasts. Each labels a balanced subset under randomized conditions. The conditions are no suggestion, correct suggestion, wrong-near suggestion (the confusable sibling), and wrong-far suggestion (a same-genus but dissimilar class). Each also labels under two interface variants: the ranked top-two list and the unranked contrastive pair. Assignment is randomized per image and per annotator, so each image appears under every condition across the pool. This design yields per-image causal contrasts.

We record accuracy against expert gold, wrong-suggestion acceptance, time per image, entropy across annotators, and a confusion-alignment score. The confusion-alignment score is the correlation between the off-diagonal human error matrix and the suggesting model's confusion matrix. A secondary arm adds the four-stage reasoning trace produced by Fine-R1 to the single suggestion. It asks whether persuasive rationales deepen near-sibling anchoring.

Downstream consequences. We will fit the acceptance model above and use it to relabel the full CUB training set. This produces confusion-aligned noise at the human-measured rate, together with a uniform-noise control at the same overall rate. On each version we train ProtoPNet, which compares image patches to learned class prototypes, and a network with the ICASC losses of Reducing Visual Confusion with Discriminative Attention, which penalizes attention overlap with the most confusing class. We then evaluate on the scratch gold. ICASC is a pointed test case. Its loss specifically targets the most-confused pair, so if that pair's labels are corrupted, the loss may push the network toward the wrong boundary.

Benchmark inflation. We also build an "assisted gold" version of the audit set from the study's single-suggestion condition. We then measure home-field advantage, following the syntax paper: how much better the suggesting model family scores on assisted gold than on scratch gold, compared with the unrelated family.

Baselines

The naive reference is unassisted from-scratch labeling under the same interface, time limit and annotator pool. It fixes the error rate each image would have without any anchor. The core claim is only interesting if wrong-near suggestions push errors above this level. No FGVC-specific from-scratch lay accuracy is reported in the file. The closest figure is the ~66% human-in-the-loop accuracy on Birds-200 from Visual Recognition with Humans in the Loop. That comes from an attribute-question system rather than label verification, so we treat it as context, not a target.

The standard practice baseline is single-label verification: show the model's top prediction and ask for confirmation or correction. This is what the syntax, pathology and video studies audit. On the same images, the contrastive pair must beat it on accuracy against gold, with no more than 15% extra time per image.

The strongest competitor is the ranked top-two list, which most tools already offer. It narrows the choice just as our design does, so it isolates our key ingredient. The ranked-versus-unranked comparison uses identical candidate pairs, identical reference images and identical time budgets. It therefore measures only the effect of revealing the model's preference. Beating it means a lower wrong-near acceptance rate and lower confusion alignment at equal accuracy on correct-suggestion trials.

The strongest existing de-biasing protocol is multi-model agreement gating from Anchoring and Agreement in Syntactic Annotations. Suggestions are shown only when the two backbones agree, and annotators label from scratch otherwise. The file gives no reported result for this protocol in vision. Its NLP benefit is framed through the 34–49% error-inflation figure it is meant to remove. We run it with the same two backbones and report accuracy, time, and the fraction of images routed to scratch.

For the downstream stage, the reference numbers are those reported for clean CUB training. ProtoPNet reports 78.0% accuracy with VGG19 (versus 75.1% for plain VGG19), and ICASC reports a +4.8% gain. We retrain both under our own compute and tuning budget on clean, uniform-noise and confusion-aligned labels, so that only the labels differ.

Why not just…?

Why not just show the top-k list, as annotation tools already do? This is the real threat. A ranked list may already dilute the anchor enough that our variant adds nothing. The ranked-versus-unranked arm is designed to settle it. If the two perform the same on wrong-near acceptance, we will report that revealing preference does not matter and drop the interface claim. The measurement contribution would stand on its own: the similarity-dependent acceptance curve and the confusion-aligned noise.

Why not just gate suggestions on model agreement or confidence? The syntax authors themselves found that different parsers share blind spots. In fine-grained vision, unrelated backbones should also agree most confidently on the very look-alike pairs where both are wrong. Gating would then suppress suggestions on easy images and keep them on the dangerous ones. Our study tests this directly: we measure wrong-near acceptance among the gated-in images.

Why not remove the human and let a reasoning model label everything? Fine-R1 improves over its base model by +8.51% in closed-world 4-shot recognition. It is still far from error-free, and its errors should also concentrate on sibling pairs. Without scratch-built gold there is no way to measure that. The rationale arm also tests whether its explanations help or hurt when shown to people.

Why not fix it in training with a confusion-aware loss? ICASC and prototype methods assume the labels on the confusable pair are right. The downstream stage checks whether confusion-aligned noise hurts them more than uniform noise at the same rate. If it does not, that is a useful negative result, and we will say so.

Decisive Checks

The central claim fails if any of three things happens. The slope $b$ could be indistinguishable from zero. Wrong-near assisted error could fail to exceed from-scratch error on the same images, which is the preregistered primary test with paired per-image contrasts. Or the confusion-alignment score of assisted errors could fail to exceed that of unassisted errors.

It would be supported by three further results. Confusion-aligned noise should lower scratch-gold accuracy of ProtoPNet and ICASC more than uniform noise at a matched rate. The suggesting model family should gain an advantage on assisted gold that the unrelated family does not share. And annotators should assign confident labels to the indeterminate stratum mainly when a suggestion is present.

Two outcomes would be scientifically informative even though they cut against us. Near-sibling acceptance might stay low, reproducing the flat entropy of the video study. That would show fine-grained verification behaves as a scaffold, and that current practice is safer than feared.