Active learning for object detection asks which images a human should label next. The usual recipe scores every predicted box by how uncertain the detector is, then gives the image the score of its single most uncertain box. This "maximum uncertainty" rule is popular because it does not let one hard object get washed out by ten easy ones. It has two known failure modes and leaves one question open. It spends labels on boxes that only look uncertain because they sit between two normal appearances of the same class, such as a sitting dog and a running dog. It also spends labels on boxes that are blurry, noisy or unlearnable. And it gives no principled answer to when labeling can stop: checking accuracy on a held-out labeled set costs about as many labels as active learning was supposed to save.
We propose one structure that addresses all three. The detector's classification head keeps several learned "prototypes" per class, meaning reference vectors that each stand for one recurring appearance mode of that class. Each predicted box is assigned to its nearest prototype, so the unlabeled pool splits into cells indexed by (class, mode). Selection takes the maximum uncertainty within each cell and sums across cells. Uncertainty counts only confusion between classes, not spread within a class. The same cells then serve as strata for a small labeled audit. The audit decides when to stop, and it marks cells whose errors persist no matter how many labels they receive so that the budget moves elsewhere. The idea in two sentences: partition detected objects into prototype cells, and query by "max within a cell, sum across cells." Then check each cell with a few audit labels, which certifies when the detector is good enough and filters out hopeless cells far more cheaply than a uniform validation set.
Core Claim
The claim has two parts and is testable. First, cell-structured acquisition should match or beat flat max, flat sum and hierarchical aggregation on PASCAL VOC and MS-COCO at equal label budgets. The largest gains should appear on scenes with many same-mode instances, where flat sum over-counts and flat max ignores additional distinct modes. Second, a stratified audit over cells can certify a target detection error using substantially fewer labels than an unstratified held-out set at the same confidence level. Cells that are clearly fine or clearly broken resolve quickly, so audit labels concentrate on cells near the threshold.
We state up front a version of the idea that would make our design smaller. The cells might be produced just as well by clustering a standard detector's features after training, with no multi-prototype head. If that holds, the contribution is the cell mechanism and the head is dispensable, and we will report it that way. The experiments include this ablation explicitly.
Where This Sits
The selection half addresses a documented disagreement in active detection.
- Max aggregation. Active Learning for Deep Object Detection via Probabilistic Modeling models box coordinates and class outputs as Gaussian mixtures and finds max aggregation strongest. Hybrid Active Learning with Uncertainty-Weighted Embeddings scales penultimate features by uncertainty, runs k-center selection, and also favors max for detection.
- Sum aggregation. Active Learning for Deep Object Detection finds that summation works better on VOC with YOLO-Small.
- Hierarchical aggregation. Active Learning for Object Detection with Evidential Deep Learning and Hierarchical Uncertainty Aggregation predicts Dirichlet evidence in one forward pass and aggregates over a hand-thresholded object, scale and class hierarchy.
- Diversity filtering. Entropy-based Active Learning for Object Detection with Progressive Diversity Constraint suppresses overlapping boxes by entropy and uses per-class prototypes (DivProto) to reject redundant images.
These papers motivate the proposal: they show that the choice of aggregation rule matters and that redundancy is the core problem. None of them groups boxes by appearance mode inside the detector's own head. DivProto is the closest work. It uses prototypes as an external diversity filter at acquisition time, keeps a single representation per class, and still sums entropy. We instead define uncertainty itself relative to a multi-mode prototype bank, and we reuse the resulting cells for verification. DivProto has no verification step.
The representation half supplies the method. SoftTriple Loss: Deep Metric Learning Without Triplet Sampling keeps K learnable centers per class and combines them with a smoothed maximum. Its sparsity penalty prunes redundant centers. It raises CUB-2011 Recall@1 from 57.8% for a single-center softmax to 60.1% at 64 dimensions. We reuse this head almost unchanged. PMAL: Open Set Recognition via Robust Prototype Mining filters prototypes by whether their neighborhoods stay stable across model initializations. We borrow that stability test to decide when cells are trustworthy enough to audit. We expect this to partly address the cold-start problem that Wu et al. report: uncertainty ranking is near-random in the first 2,000 to 3,000 images.
The verification half supplies the constraint and the motivation for auditing per cell.
- The True Sample Complexity of Active Learning shows that requiring an active learner to also certify its error reintroduces an Ω(1/ε) label cost. Without certification, the learner can beat passive learning.
- The Sample Complexity of Self-Verifying Bayesian Active Learning shows that prior structure lets a learner stop on its own with o(1/ε) expected queries. Its results apply only to simple one-dimensional classes and assume a correct prior.
- PAC Verification of Statistical Algorithms shows that verifying a hypothesis can be quadratically cheaper than learning it. For unions of d intervals, it reaches the tight bound by splitting the domain into bins and testing each bin.
We read prototype cells as a learned, high-dimensional analogue of those bins. We do not claim their bounds transfer; that would be an overclaim. These results tell us two things. Uniform certification is the expensive default to beat, and partitioning is the structure that has made verification cheap in the one case where it was worked out. On Improving Deep Active Learning with Formal Verification is the other existing bridge between selection and verification. It uses an exact solver to measure distances to the decision boundary for small classifiers, and the authors note its cost. That approach does not scale to detectors, so we verify statistically, with labels, rather than formally.
Research Program
Detector head. We replace the classification head of Faster R-CNN and RetinaNet with a SoftTriple-style head that holds K prototypes per class (K up to 8), pruned by the sparsity penalty. Each box b receives a cell a(b) = (c, k), the class c and prototype k with the highest similarity. Its uncertainty u(b) is the entropy of the class distribution obtained by taking a smoothed max over each class's prototypes. Because of the smoothed max, a box close to two prototypes of the same class has low entropy. Only confusion between classes raises u(b).
Image score. The score of an unlabeled image x is
$$s(x) = \sum_{j \in \mathcal{C}(x)} w_j \max_{b \in x:\, a(b)=j} u(b),$$
where $\mathcal{C}(x)$ is the set of cells present among the image's detections after entropy-based suppression of overlapping boxes, and $w_j \in [0,1]$ is a weight set by the audit. The weight starts at 1. It drops toward 0 when cell j is certified good, or when it has been flagged as irreducible because its audited error stayed high across rounds despite added training labels. Taking the max within a cell removes ten-copies-of-the-same-car redundancy. Summing across cells rewards images that contain several distinct unresolved modes. The audit weight replaces ad hoc trimming of outlier boxes.
Audit. Every few acquisition rounds, once cell assignments are stable across two seeds (the PMAL-style check), we spend a small audit budget. Audit images are sampled with probability tied to which cells they contain. Annotators label them fully, and we compute each cell's instance error: a missed object, a wrong class, or a box overlapping the true box by less than half its area (IoU < 0.5). Labeling is per image, so one audit image informs several cells, and we account for that in the estimator. Labeling stops once
$$\sum_j \pi_j \, \mathrm{UCB}_j \le \epsilon,$$
where $\pi_j$ is the estimated fraction of pool instances in cell j, $\mathrm{UCB}_j$ is an upper confidence bound on that cell's error (Clopper–Pearson with a union-bound correction over cells), and ε is the target error. Audit labels count against the total budget. Audited images are kept out of training until the decision to stop, so the error estimates stay unbiased, and are added to training afterward.
Analysis. We will measure how total audit cost grows with the number of non-pruned cells and with ε. Uniform certification predicts roughly 1/ε growth regardless of structure. If stratification helps, audit cost should grow more slowly, and the cells near the threshold should account for most of it. Whether the growth looks closer to √(cells), as the interval result might suggest, is a question we will measure, not assume.
Baselines
Random selection is the naive reference point. It uses the same detector, training schedule and per-round budget, with images drawn uniformly. Every method must beat it, and the early rounds show whether our stability gating actually helps with cold start.
Flat-aggregation single-center baselines. These are max, sum and mean aggregation of entropy on the unchanged single-center detector, as in Choi et al. and Brust et al. They isolate whether anything beyond the choice of aggregation rule matters.
Strongest existing methods. These are DivProto with entropy-based suppression from Wu et al. and evidential hierarchical aggregation from Park et al.. DivProto is the closest competitor because it already combines uncertainty, redundancy suppression and class prototypes. It reports 73.68% mAP on VOC with 7,000 images and 32.87% AP on COCO with 40% of the training data using Faster R-CNN, 1.25 points above random. Park et al. report about 74% mAP on VOC with 20% of the labels, against 66–67% for entropy and random sampling. Choi et al. report 75.98% mAP on VOC with 10,000 images. UWE has no reported numerical VOC result on Lacuna, and the evidential method has no reported COCO number on Lacuna.
These figures come from differing detectors and initial pools, so we will rerun every method ourselves. All runs share the same backbone, initial labeled set, per-round budget, training epochs, three seeds, and an equal hyperparameter search budget per method, about 10 configurations tuned on a held-out VOC split. We also run UWE with k-center selection as the strongest combined uncertainty-and-diversity baseline.
What counts as winning on selection. Beating DivProto means a higher area under the mAP-versus-labels curve across all seeds, or reaching the 73.68% VOC level with clearly fewer images in our reproduction. A gain within seed spread does not count.
Stopping baselines. For stopping, the reference is an unstratified held-out validation set sized to give the same confidence on the same ε. Our audit wins if it stops at a true test error within ε while spending fewer audit labels.
Ablations. Four ablations isolate the key ingredients:
- Single-center head with cell-structured scoring, using cells from k-means on that detector's features.
- Multi-prototype head with flat max scoring.
- Cell scoring with all $w_j$ fixed at 1, that is, no audit feedback.
- Uniform rather than stratified audit sampling.
Why not just…?
Why not just cluster an ordinary detector's features instead of training a multi-prototype head? This is the most serious threat, and we cannot rule it out on paper. Post-hoc k-means yields cells too. The case for the learned head is that its prototypes are trained so that within-class spread does not register as class confusion. SoftTriple's gain over a single center on fine-grained retrieval suggests these modes are real and learnable. Ablation 1 settles it. If clustering a single-center detector matches the head, we report cell-structured selection and auditing as the contribution and recommend the cheaper clustering.
Why not just use DivProto or uncertainty-weighted k-center, which already trade off uncertainty against diversity? Both enforce diversity across images by rejecting candidates near already-labeled examples. Neither removes redundancy within an image or separates within-class spread from between-class confusion, and neither tells you when to stop. The head-to-head comparison above decides the selection question. The stopping question is something these methods do not address at all.
Why not just trim outliers, for example by averaging the top three box uncertainties instead of taking the max? Trimming suppresses isolated noisy boxes, but it equally suppresses a single, genuinely informative rare object. Audit flagging instead removes only cells whose errors persist after they receive labels. We include top-3 mean as a baseline. We will also measure what fraction of each method's budget goes to images whose selected boxes annotators mark as unlabelable or ambiguous.
Why not just evaluate on a held-out validation set each round? Balcan et al. and Yang et al. show this costs on the order of 1/ε labels and can erase the active-learning savings. It is nonetheless simple and reliable, so it is our stopping baseline. If stratification does not cut audit cost meaningfully, the verification half of the proposal fails, and we will say so.
Proposed Experiments and Decisive Checks
Benchmarks. We use PASCAL VOC 2007+2012 and MS-COCO with Faster R-CNN and RetinaNet, following the per-round protocols of Wu et al. and Choi et al. We add the SelvaBox tree-crown detection dataset as a stress test, because its densely packed, same-mode instances are where flat sum should over-count and flat max should under-count. No reported result was found on Lacuna for any of these active learning methods on SelvaBox.
Metrics. These are mAP (or COCO AP) against the number of labeled images, the area under that curve, and images needed to reach fixed mAP targets. For stopping, we record the true test error at the moment of stopping relative to ε, total audit labels, and the false-certification rate across seeds. For the outlier question, we record the share of budget spent on images later flagged as irreducible.
Decisive checks. Three checks determine the verdict.
- The cell rule should win most clearly on SelvaBox and on COCO images with many instances. If its advantage is uniform across scene density, the max-within/sum-across mechanism is not what drives it.
- Ablation 1 must show whether the learned head or the cells themselves carry the gain.
- Across ε values from 0.30 down to 0.15 detection error at IoU 0.5, the stratified audit must certify at least as reliably as an equal-confidence uniform validation set, with materially fewer labels. Its cost must also grow more slowly as ε shrinks.
If the third check fails, the cells still have value for selection. In that case, the claimed link to the verification–learning gap would not hold.
References
- Active Learning for Deep Object Detection via Probabilistic Modeling
- Hybrid Active Learning with Uncertainty-Weighted Embeddings
- Active Learning for Deep Object Detection
- Active Learning for Object Detection with Evidential Deep Learning and Hierarchical Uncertainty Aggregation
- Entropy-based Active Learning for Object Detection with Progressive Diversity Constraint
- SoftTriple Loss: Deep Metric Learning Without Triplet Sampling
- PMAL: Open Set Recognition via Robust Prototype Mining
- The True Sample Complexity of Active Learning
- The Sample Complexity of Self-Verifying Bayesian Active Learning
- PAC Verification of Statistical Algorithms
- On Improving Deep Active Learning with Formal Verification