Let the Editor's Corrections Steer Prototype Selection

When a model drafts labels and a human corrects them, the expensive part is the correcting, and it is spent on whichever images the system shows next. Methods that pick a few "typical" examples to label first work very well when people label from scratch. They choose points at dense spots in a learned feature space, the most common-looking examples. The proposal here is to keep that dense-prototype starting point but let each human correction change what gets picked next. The size of a correction is a direct, model-independent measurement of how wrong the drafts are in that part of the data. That measurement can be spread to similar examples in the same feature space. It can then decide two things: which neighborhoods still need a human, and which can have their drafts accepted without review.

This matters because the two literatures currently pull in opposite directions. Low-budget active learning says to query typical points, because early models cannot locate their own errors. Model-assisted labeling says human effort pays off most where drafts are wrong. Corrections observed at prototypes reconcile the two. They are available from the very first query, so the system can begin with typical examples and move toward representative error pockets as the edits accumulate. It does not need a warmed-up model or a hand-chosen moment to switch strategies.

Core Claim

In draft-and-edit annotation, a selection rule that weights density-based coverage by draft error estimated from observed human edits will reach a target downstream accuracy at lower total editing cost than three alternatives: density selection alone, foundation-model confidence, and class-imbalance acquisition. The same estimate supports a safer auto-accept rule for unreviewed drafts.

The evidence for the starting point comes from the density work. Active Learning on a Budget introduced TypiClust, which clusters self-supervised features and queries the densest point of each unlabeled cluster. It showed that even oracle uncertainty underperforms random sampling at tiny budgets. Active Learning Through a Covering Lens (ProbCover) and Generalized Coverage (MaxHerding) recast the same goal as covering the data with small, label-pure neighborhoods, hard balls in the first case and soft kernels in the second. We reuse MaxHerding's soft-coverage greedy as the base mechanism.

All three methods assume that a queried point is labeled from scratch at a fixed cost. COLosSAL makes the same assumption when it benchmarks cold-start selection on Medical Segmentation Decathlon tasks. It also reports the key failure: no cold-start method reliably captured small, irregular tumors. That failure motivates our proposal. Tumors do not form dense clusters, so pure density selection never visits them, but drafts on them are badly wrong and the edits would say so.

The closest prior idea is REAL. REAL allocates labeling budget to clusters with many "pseudo errors," meaning points whose prediction disagrees with their cluster's majority prediction. It needs roughly 0.80 pseudo-label accuracy before this works, and it handles only categorical labels. Our error signal is the human's actual correction to a structured output, so it exists from query one.

Research Program

Each unit (an image, a volume, or an object mask) has a self-supervised embedding $x_i$ and a draft from a foundation or seed model. When the human edits unit $j$, we record the edit magnitude $e_j = 1 - \mathrm{Dice}(\text{draft}_j, \text{corrected}_j)$, which is zero if the draft was accepted unchanged. We also record the editing cost $c_j$, measured in clicks or seconds.

For any unit, the estimated draft error is a kernel-smoothed average of the edits observed so far:

$$\hat r(x) = \frac{\alpha r_0 + \sum_{j\in S} k(x,x_j)\, e_j}{\alpha + \sum_{j\in S} k(x,x_j)}$$

Here $S$ is the set of edited units and $k$ is the MaxHerding kernel. The prior error $r_0$ is shrunk toward local evidence with strength $\alpha$. The estimated cost $\hat c(x)$ is smoothed the same way.

An edited prototype covers its neighbors only to the extent that its draft was reliable. A large edit signals that nearby drafts are probably also wrong, so the region should not count as handled. We therefore define the current coverage of unit $i$ as $m_i = \max_{s\in S} (1-e_s)\,k(x_i,x_s)$. The next query is then

$$j^* = \arg\max_j \frac{1}{\hat c(x_j)} \sum_i \hat r(x_i)\,\big[\max\big(k(x_i,x_j),\, m_i\big) - m_i\big].$$

Before any edits, $\hat r$ is the constant $r_0$ and $\hat c$ is uniform. The rule then reduces exactly to MaxHerding and selects typical points. As edits arrive, neighborhoods with small edits become covered and stop attracting queries. Neighborhoods with large edits stay uncovered and carry high weight, so the rule keeps sampling them. Because the gain still sums over neighbors, it favors representative points inside an error pocket rather than isolated outliers.

Between rounds, the drafting model is fine-tuned on the corrected labels and redrafts the pool. Old edit magnitudes then overstate the current error, so we age them out and recompute $\hat r$.

A unit that is never queried is accepted as a training label only if two conditions hold: its coverage $m_i$ exceeds a threshold $\tau$, and its estimated error $\hat r_i$ is below $\epsilon$. Otherwise it is excluded from training. This replaces the learned correction-propagation network of A2LC with a transparent geometric rule, which we can audit for poisoning.

Human editing is simulated by an automatic clicker. It places a corrective click at the largest error region until the Dice score reaches a target, using an interactive segmentation model in the style of You Point, I Learn, and cost is the number of clicks. A small study with real annotators on fundus images will check whether click counts track real time.

The benchmarks are chosen so each connects to a published workflow. The first is the five COLosSAL tasks of The Medical Segmentation Decathlon, especially Liver & Tumor and Pancreas & Tumor. The second is Cityscapes and PASCAL VOC 2012 at the mask level, following A2LC. The third is the fundus and BraTS shift settings of You Point, I Learn.

Baselines

The naive reference points come first. One is to do nothing: train on the foundation-model drafts with no human edits. The other is random selection of drafts for editing, run over many seeds. COLosSAL found random to be a strong but high-variance baseline on large organs.

Density selection alone means TypiClust and MaxHerding choosing drafts for editing with no edit feedback. It is the most direct competitor and the obvious combination of the two directions. It differs from our method only in the error weighting, so it doubles as the key ablation. We beat it if our method reaches the target Dice or mIoU with clearly fewer clicks, in particular on the tumor tasks where COLosSAL reports density methods fail. COLosSAL gives no numeric task-level Dice; there is no reported result found on Lacuna for TypiClust's per-task score on the MSD cold start, so we will run it ourselves.

The strongest existing system is A2LC: Grounded SAM drafts, acquisition that balances classes by KL divergence, and a learned correction network that propagates edits. It reports "27.23% mIoU gain over the baseline under the same budget" on Cityscapes and "94-95% of fully supervised performance" with "20% of the budget" on Cityscapes and "60%" on PASCAL VOC (A2LC). We will run it on the same masks, drafts, and click simulator, with the same budget for tuning hyperparameters. Beating it means reaching the same fraction of fully supervised mIoU at a lower budget, or a higher fraction at equal budget.

A confidence-weighted variant replaces the edit-based $\hat r$ with one minus the foundation model's own predicted mask quality, keeping the coverage term. This ablation isolates whether human edits carry information that model confidence does not. Two further ablations separate the selection rule from the propagation rule. The first uses our selection with A2LC's correction network. The second uses A2LC's selection with our coverage-based auto-accept.

Why not just…?

Why not just run TypiClust or MaxHerding on the drafts? This is the reviewer's default, and it may well win on large, homogeneous organs where drafts are uniformly good. It ignores what each edit reveals about its neighborhood. COLosSAL shows that density selection misses tumors. Its prototypes also risk being "rubber-stamped" when drafts on typical points are already accurate, which wastes expensive edits. The edit-feedback ablation settles the question directly: if the gap closes on the tumor tasks, the idea adds nothing there.

Why not just rank drafts by the foundation model's own confidence, weighted by density? This is the real threat. It costs nothing extra and needs no edits. Our doubt rests on You Point, I Learn, which documents that zero-shot models fail even on typical anatomy under scanner and pathology shift, a gap of more than 10% in Dice. A model's confidence is least trustworthy in exactly that setting. The confidence-weighted ablation is run under both shifted and unshifted conditions. If it matches our method under shift, we will report that model confidence suffices and that edits matter only for deciding what to auto-accept.

Why not just use uncertainty sampling or Coreset? Active Learning on a Budget shows that both do no better than random sampling at one to five labels per class. Uncertainty sampling fails even with oracle uncertainty. Coreset chases distant outliers at small budgets, as Active Learning Through a Covering Lens explains. We include margin sampling once as a sanity check rather than as a main competitor.

Why not just use REAL's error-density allocation? REAL needs about 80% pseudo-label accuracy before its majority-rule error proxy means anything. It also defines errors only for categorical labels. We will run REAL on the classification-style mask units after a warm-up to test whether human edit magnitudes outperform its pseudo-errors once both are available.

Proposed Experiments

The first check is a gate that runs before anything else. On every benchmark, measure whether edit magnitudes are smooth in embedding space: does the edit at a prototype predict edits at its kernel neighbors better than the global mean does? If they are not smooth, $\hat r$ is noise and the proposal fails at the source. We would then report that negative result.

The main experiment plots editing cost against downstream Dice or mIoU for all baselines over many seeds. It reports cost to reach the target, the area under the curve, and worst-seed performance, since COLosSAL's main argument for density methods was avoiding catastrophic draws.

A behavior analysis tracks the fraction of queries that land in high-density regions versus error pockets over rounds. The method should begin like TypiClust and drift toward tumors or tail classes. This gives an empirical, data-driven answer to the open question in Active Learning on a Budget of when to switch from typical to hard examples.

A safety check measures the precision of auto-accepted labels against ground truth and the downstream damage of injecting errors. It compares our coverage rule against A2LC's correction network. It also repeats the comparison with a simulated fatigued annotator who accepts mediocre drafts, the confirmation-bias failure mode flagged in You Point, I Learn.

Finally, the real-annotator fundus study checks that the ranking of methods under the click simulator holds up when cost is measured in actual time.