Local-versus-context disagreement as a gate for labels and memory in segmentation

Segmentation models that keep an explicit memory of stored pixel features can predict a pixel's label by looking up its nearest stored neighbors. That memory is expensive, because it grows with every patch ever seen. The same models are also expensive to label, because annotating whole images wastes effort on regions the model already gets right. This proposal treats both costs as one problem. We pair two predictors inside a single network. The first is a local predictor: a nearest-neighbor lookup over a memory bank of features, each describing only a small image neighborhood. The second is a context predictor: a head that integrates information across the whole image. Where the two agree, local appearance already settles the label, so the region needs neither a human label nor a memory slot. Where they disagree, a patch looks like one class but its surroundings say another. A local memory will get exactly those patches wrong, and exactly those patches are worth an annotator's time.

The proposal is to use this built-in disagreement as the only rule for two decisions. It decides which image regions to send for labeling, and it decides which labeled features get written into, or evicted from, the memory bank. The testable claim is that memory size then grows with the number of context-dependent regions rather than with the total number of pixels, and that label budget goes to the regions where a limited field of view actually fails. Both should hold at matched accuracy and without paying for dropout sampling or model ensembles.

Core Claim

Active learning selects which unlabeled examples a human should label next. Methods that select by disagreement usually build a committee of near-identical hypotheses. Examples include repeated forward passes with random dropout (Bayesian Active Learning by Disagreement, or BALD, as studied in A Study of Acquisition Functions for Medical Imaging Deep Active Learning) and two adversarially trained classifier heads on a shared backbone (Agreement-Discrepancy-Selection). In both cases the committee members see the same receptive field, so their disagreement measures generic parameter uncertainty.

We propose a committee whose members differ in what they can see. The memory lookup only sees local texture. The context head, a recurrent criss-cross attention module from CCNet, reaches every pixel at about $O(N\sqrt{N})$ cost instead of the quadratic cost of dense non-local attention. Their disagreement targets a specific failure: a patch that is locally ambiguous and resolved only by distant context. Guanbin Li et al. argue that locality prevents CNNs from resolving textures that depend on distant anchors. We turn that weakness into an acquisition signal.

The memory is a direct beneficiary of this signal. A local-feature bank is exactly where "sky" and "water" patches with the same texture collide. It is also where current pruning is blind: coreset subsampling cut the patch bank in Is Task-Specific Training Necessary for Anomaly Detection? from 32.6 GB to 0.37 GB, but it chooses entries by geometric coverage rather than by whether they matter for decisions.

For a region $r$ (a superpixel or small patch cell), the disagreement score is

$$D(r) = \frac{1}{|r|}\sum_{i \in r} \mathrm{JS}\big(p_{\text{mem}}(y \mid x_i),\; p_{\text{ctx}}(y \mid x_i)\big),$$

where $p_{\text{mem}}$ is the class distribution from similarity-weighted nearest neighbors in the memory bank, $p_{\text{ctx}}$ is the softmax of the context head, and JS is the Jensen–Shannon divergence. The score costs one forward pass plus one memory lookup per candidate.

Research Program

The system has a shared ResNet backbone, as in CCNet. Local features are read from an intermediate stage, and we will measure that stage's effective receptive field (the region that actually influences a feature, as opposed to the theoretical maximum) to confirm it is local rather than assume so. The context head is a two-loop criss-cross attention module on top of the backbone. The memory bank stores labeled local features with their labels. This resembles the class-wise bank of Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank, which fills per-class first-in-first-out queues by prediction confidence. Our bank differs in how entries arrive and leave.

Each active round has three steps. First, unlabeled images are scored region by region with $D(r)$. Candidate regions are then filtered for redundancy with a retrieval check: a candidate is skipped if its pooled feature lies within a radius of a region already chosen this round. We borrow this idea of spatially constrained, retrieval-based comparison from RAD. It targets the documented failure in Revisiting Deep Active Learning for Semantic Segmentation, where uncertainty-based selection kept picking near-identical frames from redundant driving video.

Second, the selected regions are labeled. Their local features are written to memory only if the memory's prediction was wrong or $D(r)$ exceeded a threshold. Consensus features are never written, because the context head and the existing neighbors already cover them.

Third, when the bank reaches its budget, we evict entries that have not been retrieved as a neighbor inside any high-disagreement region over recent rounds. After each round the backbone is fine-tuned, and stored features are re-encoded at that point. Because the bank is small, a full refresh is affordable, which sidesteps the stale-feature problem that forces large banks into queues.

We then analyse where disagreement lands. We check whether it concentrates on context-dependent classes and small structures or simply on object edges. The likely weakness of this signal is that both predictors disagree at every boundary because of label ambiguity there. We will report results with and without masking a thin band around predicted edges.

Baselines

The naive reference is random region selection at the same labeled-pixel budget, with a memory bank that stores every labeled patch and no eviction. A second naive point is predictive entropy of the context head per region, which is the cheapest single-model score. Revisiting Deep Active Learning for Semantic Segmentation reports that entropy reaches 60.16 mIoU on A2D2-3K against 58.93 for uniform sampling, so entropy is a real, not strawman, reference.

The strongest acquisition competitor is region-level BALD computed with Monte Carlo dropout on the same hybrid network and memory. It asks whether stochastic committees already capture what our structured committee captures. To make the comparison fair, it gets the same backbone, schedule, and tuning budget. We report it at 10 and 20 dropout passes, with its extra compute stated. We also report our method given matched wall-clock time.

The strongest published image-level method on redundant driving data is Coreset combined with semi-supervised learning. It reaches 61.13 mIoU on A2D2-3K (Revisiting Deep Active Learning for Semantic Segmentation). We rerun it under our pixel-count budget accounting, and beating it means higher mIoU at an equal number of labeled pixels. For region-level active learning on Cityscapes we found no reported result on Lacuna. There, CCNet at 81.9% test mIoU with full labels serves as the ceiling. The class-wise memory bank's 59.4% mIoU at 1/30 of labels serves as a passive semi-supervised reference point.

On the memory side, the competitor is a bank holding all labeled patches, pruned to our memory budget by greedy coreset as in RAD, and by first-in-first-out queues as in the class-wise memory bank. Beating it means equal or better mIoU on rare and context-dependent classes at the same number of stored vectors.

The ablation that isolates the key ingredient replaces the context head with a second head that has the same local receptive field. This is an Agreement-Discrepancy-style pair of classifiers with homogeneous views. Everything else stays fixed. If this pair matches our method, then differing receptive fields add nothing and the central claim fails.

Why not just…?

Why not just run BALD with MC dropout on the hybrid network? This is the most serious threat. BALD beat entropy and MeanSTD on ISIC 2016 (accuracy 0.8047) in A Study of Acquisition Functions for Medical Imaging Deep Active Learning. Dropout disagreement, however, is not tied to receptive field. It cannot tell the memory which entries will collide, and it multiplies inference cost by the number of passes. The matched-compute comparison and the correlation analysis settle it. If BALD's per-region score predicts memory-lookup errors as well as $D(r)$ does, we will adopt BALD as the gate and drop the structural-committee claim.

Why not just keep a full bank and prune it with coreset, and pick labels with entropy separately? Coreset kept RAD's anomaly scores intact at under 2% of the storage. That task scores closeness to normal data and has no class boundaries to preserve. The concern is that geometric coverage underweights rare boundary regions. Agreement-Discrepancy-Selection reports that Coreset picks outliers far from decision boundaries when labels are scarce. The memory-versus-mIoU curve, broken down per class, tests this directly.

Why not just drop the memory and use a larger context model? The parametric-only variant is part of the ablation grid. ConvFormer documents that pure transformers need pre-training or convolutional stems on small data. The class-wise memory bank reports a +5.3 mIoU gain over ClassMix at 1/30 labels from anchoring features to a bank. If the memory contributes nothing at our budgets, the labeling gate still stands on its own, but the memory half of the claim does not.

Why not just predict segmentation accuracy with an auxiliary network, as PAAL does? Predictive Accuracy-Based Active Learning for Medical Image Segmentation needs a separate ResNet-18 trained to predict Dice scores, plus plateau-triggered querying, and it scores whole slices. We include it on the medical transfer experiment using its released code (shijun18/PAAL-MedSeg). It reports relative gains only, for example +4.9% Dice over MaxEntropy at a 5% budget on the MSD brain tumour data, so we compare against our own rerun rather than a quoted absolute number.

Proposed Experiments

The primary benchmarks are Cityscapes and the redundant A2D2 pools, including A2D2-3K. Budgets are counted in labeled pixels, and memory is counted in stored vectors and gigabytes. We report mIoU against labeled pixels, mIoU against memory size, per-class IoU for context-dependent and rare classes, and query latency. PASCAL VOC 2012 serves as a less redundant check. The MSD brain tumour data, with SegTHOR as a second medical set, tests transfer to medical segmentation, where local texture is often misleading and anatomical context decides the label.

Three results would be decisive. The first is the correlation between each score ($D(r)$, entropy, BALD) and the memory lookup's actual per-region error; $D(r)$ must correlate best for the gating logic to hold. The second is the same-receptive-field ablation, which must lose clearly to the true local/context pair. The third is the growth of the memory bank over rounds: under disagreement gating it should flatten while mIoU keeps rising, whereas coreset- and queue-managed banks of equal size should lose accuracy on context-dependent classes first. If disagreement mostly lands on object edges and edge masking erases the gain, we will conclude that the signal measures boundary ambiguity rather than a limited field of view, and report that outcome.