A standard image classifier often averages its evidence over the whole image before deciding. Many networks do this explicitly: they compute class scores at every spatial location and then take the mean. Averaging hides conflict. If the bird's beak strongly says "species A" and the surrounding foliage strongly says "species B", the two votes cancel, and the image-level prediction looks merely uncertain. The same averaging can also hide a confident mistake, when one strong spurious patch outvotes the true but weaker diagnostic cue. Counting research has a name for this failure, error cancellation, where background over-counts offset foreground misses so that the total still looks right. Active learning is the practice of choosing which unlabeled images to send to a human for labeling. Its usual scores, such as entropy or the gap between the top two class probabilities, are computed after this averaging. They therefore cannot tell an image whose patches strongly disagree from an image whose patches carry no evidence at all.
We propose a fix that adds almost no machinery, and we test it where this distinction matters most: fine-grained recognition, where categories such as bird species, car models, or aircraft variants differ only in small local details. For the two classes the model finds most confusable, we compute a per-patch preference. The acquisition score is the amount of that preference that is cancelled when patches are averaged. Our claim is that images with high cancelled evidence are the ones worth labeling. They mark places where the model has learned features that fire for both look-alike classes, often from background, and a label can correct that. Images whose uncertainty comes from uniformly weak evidence, such as an occluded or blurred diagnostic part, are a poorer use of the labeling budget.
Core Claim
Take a classifier whose image-level score is the spatial mean of patch-level scores. This holds exactly for a ResNet with global average pooling and a linear head, and for a Vision Transformer whose patch tokens are mean-pooled before the linear head. Let $z_{p,c}$ be the score of class $c$ at patch $p$, with $P$ patches. Let $a$ and $b$ be the top two classes of the image-level prediction, and let $m_p = z_{p,a} - z_{p,b}$ be the local preference for $a$ over $b$. The usual margin is the average $\bar m = \frac{1}{P}\sum_p m_p$. We score each unlabeled image $x$ by its cancelled evidence:
$$ C(x) = \frac{1}{P}\sum_{p} |m_p| \;-\; \Big|\frac{1}{P}\sum_{p} m_p\Big| . $$
$C$ is zero exactly when all patches agree on which of the two classes they prefer. It grows with the total local evidence that averaging throws away. The global margin splits into two parts: evidence magnitude $E = \frac{1}{P}\sum_p |m_p|$ minus cancelled evidence $C$. Margin sampling looks only at the difference $E - C$, so it treats "strong conflict" and "no evidence" as the same thing. Our hypothesis is that, under low inter-class variance, high $C$ points to learnable confusion (epistemic uncertainty, which more labels can reduce), while low $E$ points mostly to missing visual evidence (aleatoric uncertainty, which labels cannot fix). We also expect high $C$ to flag confidently wrong images that margin sampling never queries.
This is the counting-world insight carried over into an acquisition function for categorical prediction. Towards Locally Consistent Object Counting with Constrained Multi-stage Convolutional Neural Networks penalizes per-cell counting errors so that global counts cannot hide local errors. Global Sum Pooling identifies "patchwise cancellation" under average pooling. Both are training-time fixes for single-class density regression. We do not change training. Instead, we treat cancellation as a measurement, made on unlabeled data, of where the model is internally inconsistent between two specific confusable classes.
Research Program
The work has three parts.
Score and its variants. We implement $C$ and its variants. The two classes can be the top two predicted classes, or pairs taken from the training-set confusion matrix in the spirit of the confusable-class clustering in HD-CNN. A probability-weighted version sums over the top three or four classes. Because $C$ scales with logit magnitude, we rank images within each acquisition round rather than thresholding raw values.
Do high-$C$ images contain the error type we claim? We check this before any active learning loop. On a model trained on a small labeled seed, we measure how $C$, margin, entropy, and ensemble disagreement each predict which unlabeled images are misclassified. We give special attention to confidently misclassified images. On CUB-200-2011 we also measure where the conflicting patches fall relative to the bird, using the dataset's own part and foreground annotations. This tests whether high-$C$ conflict is disproportionately background-versus-part.
The active learning loop. We run the loop at matched budgets. The sharpest test is the museum-to-wild butterfly and plant setting of Domain Adaptation and Active Learning for Fine-Grained Recognition in the Field of Biodiversity. There, clutter in the wild images is expected to supply exactly the spurious patch evidence the score is designed to expose.
The decomposition is exact only for mean-pooled linear heads. Max-pooled part detectors such as DFL-CNN avoid cancellation by discarding all but the strongest patch. We treat them as an architectural contrast, not a target. One prediction is that cancelled evidence measured on a mean-pooled model still selects images that help the overall pipeline.
The prior work plays different roles. Counting papers motivate the measurement. Evaluating Crowd Density Estimators via Their Uncertainty Bounds shows that scalar summaries of dropout ensembles hide local failures, and that pixel-wise ignorance maps reveal them. It uses those maps only for evaluation, never to select data. Calibrating Uncertainty for Semi-Supervised Crowd Counting learns patch-level uncertainty, but uses it to accept pseudo-labels in single-class counting, not to query humans. Sub-image active learning constrains the design. Towards Fine-grained Sampling for Active Learning in Object Detection and PixelPick show that localized uncertainty saves labels. Both, however, need region-level labels (boxes or pixels) that fine-grained classification does not have. We keep the image-level label and use the local signal only to rank images. Finally, Privacy-preserving and Uncertainty-aware Federated Trajectory Prediction and Active Data Curation Effectively Distills Large-Scale Multimodal Models supply the warning that maximal uncertainty often selects noise. The first finds median-uncertainty selection beats the extremes. The second keeps only samples a reference model finds clean. Our evidence-magnitude term is the within-model analogue of that filter.
Baselines
Random acquisition. This is the do-nothing reference. Every method starts from the same class-balanced seed of two labeled images per class. Each acquisition round adds the same number of images. The retraining schedule, augmentation, and backbone (ImageNet-pretrained ResNet-50 at 448 pixels, and a mean-pooled ViT-B) are identical across methods, and results average five seeds. Our method must exceed random on area under the accuracy-versus-labels curve at every budget, not only at one.
Image-level margin sampling. This is the closest single-model competitor and the quantity our score explicitly splits apart. Margin sampling is cheap, strong, and widely used, so it shares our backbone and needs no extra passes. Beating it means a higher learning-curve area and fewer labels to reach 90% of the fully supervised accuracy. On CUB-200-2011, Stanford Cars, and FGVC-Aircraft, the fully supervised reference points in the file are the 87.4%, 93.8%, and 92.0% that Learning a Discriminative Filter Bank Within a CNN for Fine-Grained Recognition reports for its ResNet-50 model. These are ceilings, not active learning results. We found no reported active learning result on Lacuna for any of these three datasets, so we will establish the margin, entropy, and ensemble baselines ourselves.
Ensemble disagreement (BALD). This is the strongest existing estimate of epistemic uncertainty. BALD (Bayesian Active Learning by Disagreement) is the mutual information between prediction and model, computed over a five-member deep ensemble and, separately, over Monte Carlo dropout. Because it costs five times the training compute, we also report a compute-matched comparison: a five-member ensemble whose members each use our score, averaged.
Prior methods in the domain-shift setting. On the butterfly and plant benchmarks we reproduce the DivDis and expected-model-output-change strategies of Gruner et al. on top of their adversarially adapted model. The file reports that model at 79.87% on butterflies and 23.54% on plants, against an 81.23% plant upper bound. It does not report their per-budget active learning curves, so we rerun them.
Ablations. These isolate the key ingredient. We compare ranking by evidence magnitude $E$ alone, by the normalized cancellation ratio $C/E$ alone, by $C$ computed over all classes rather than the confusable pair, and by mean and max per-patch entropy.
Why not just…?
Why not just use margin or entropy on a strong pretrained Vision Transformer? A stronger backbone shrinks cancellation only if it stops attending to background. The file's evidence suggests it does not, at least under shift. Gruner et al. find that whole-image selection on wild images stays far below the ceiling. Background texture leakage is a documented failure of fine-grained classifiers. We run every method on a ViT-B as well as a ResNet-50. If the gain over margin disappears on the ViT, that is a real negative result, and we will report it as such.
Why not just use per-patch entropy or an ignorance map, as in the counting work? This is the most serious alternative, because it also looks before averaging. Per-patch entropy, however, measures uncertainty within each patch. A patch that confidently prefers class B next to one that confidently prefers class A has low entropy at both locations. Conflict between patches is exactly what $C$ captures and patch entropy misses. The mean and max patch-entropy ablations settle the question directly. If either matches $C$, the cancellation framing adds nothing, and we will say so.
Why not just use BALD with an ensemble, since it already targets epistemic uncertainty? BALD may well capture much of the same signal. We will measure the rank correlation between $C$ and BALD on the unlabeled pool. If the correlation is high and BALD matches $C$ on the learning curves, then $C$'s contribution reduces to a single-model, five-times-cheaper approximation. That is still useful but a smaller claim. If $C$ and BALD are only weakly correlated and their combination beats either one, the two signals are distinct.
Why not just train with sum pooling or a grid loss so cancellation never arises? These are training regularizers for counting. They do not rank unlabeled data. Summing class evidence over all patches also accumulates background in a classification setting. Zhao et al. further report that grid-loss weight must be tuned carefully, or performance degrades. We do not pursue them as competitors, though $C$ could be combined with them.
Proposed Experiments
The first decisive check is diagnostic and needs no loop. On held-out unlabeled CUB and Stanford Cars images, a seed model is evaluated by the area under the precision-recall curve for detecting its own errors. Here $C$ should beat margin and entropy specifically on images the model gets wrong with high confidence. On CUB, high-$C$ images should show conflicting patches outside the annotated bird foreground more often than low-margin, low-$E$ images do. If neither holds, the mechanism is wrong, and the loop results would be uninterpretable.
The second check is the active learning comparison. We run ten rounds on CUB-200-2011, Stanford Cars, and FGVC-Aircraft, and five rounds on the butterfly and plant domain-shift benchmarks starting from the adapted model. We report the area under the learning curve, labels needed to reach 90% of supervised accuracy, and per-round fraction of queried images that were misclassified. The prediction is that the advantage of $C$ over margin is largest in the cluttered wild-image target domains, and smallest on Aircraft, where backgrounds are mostly sky.
The third check tests the aleatoric claim. Among images with small global margin, we label one subset with high $C$ and another with low $E$, at equal budget. The model retrained on the high-$C$ subset should improve more than the one retrained on the low-$E$ subset. If it does not, the evidence-versus-conflict split is not doing the work we claim.
References
- Towards Locally Consistent Object Counting with Constrained Multi-stage Convolutional Neural Networks
- Global Sum Pooling: A Generalization Trick for Object Counting with Small Datasets of Large Images
- HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition
- Domain Adaptation and Active Learning for Fine-Grained Recognition in the Field of Biodiversity
- Learning a Discriminative Filter Bank Within a CNN for Fine-Grained Recognition
- Evaluating Crowd Density Estimators via Their Uncertainty Bounds
- Calibrating Uncertainty for Semi-Supervised Crowd Counting
- Towards Fine-grained Sampling for Active Learning in Object Detection
- All you need are a few pixels: semantic segmentation with PixelPick
- Privacy-Preserving and Uncertainty-Aware Federated Trajectory Prediction for Connected Autonomous Vehicles
- Active Data Curation Effectively Distills Large-Scale Multimodal Models