Semi-supervised segmentation trains a pixel labeler from a few annotated images and many unannotated ones. It works by having the model label the unannotated images itself. These self-made labels are called pseudo-labels, and a pixel that is labeled confidently but wrongly gets reinforced rather than corrected. Vision transformers make the problem worse at high resolution. Their self-attention compares every image patch with every other patch, so cost grows with the square of the number of patches. One effective fix is early exit: a patch whose prediction is already confident stops being processed in deeper layers. In a semi-supervised loop, though, early exit is a trap. A confidently wrong patch exits, its wrong label is frozen, and the model never revisits it. The same confidence signal that saves compute also locks in errors.
We propose that the decision to let a patch exit should come from a judge outside the segmentation model's own softmax. The judge is a small discriminator, a network trained to tell real annotation maps from predicted ones, and it scores each patch by how plausible its predicted local label structure looks. The catch is that such a discriminator, trained on a few dozen or a few hundred annotated maps, memorizes them quickly. When it does, its scores become worse than useless. So we borrow the overfitting controls developed for low-data GANs and use them to keep the discriminator honest. The single per-patch score then does two jobs: it weights each patch's pseudo-label, and it decides which patches leave the computation during training.
Core Claim
Our claim has three parts. First, using the network's own confidence for training-time early exit amplifies confirmation bias in semi-supervised segmentation: the error rate among exited patches should rise over training. Second, gating exits with a structure-aware discriminator lowers that error. Third, this only holds if the discriminator's overfitting is actively controlled. An unregulated discriminator should gate worse than plain softmax confidence, and the size of that degradation should track a measurable overfitting gap. If all three parts hold, they give something no prior method offers: real training-time compute savings in semi-supervised segmentation of high-resolution scenes, without paying for it in pseudo-label quality.
The three directions play different roles here. Discriminator overfitting is the core mechanism and the main risk. Pseudo-label noise is the application setting and the quantity we measure. Quadratic attention cost is the constraint that makes early exit worth doing at all.
What Prior Work Establishes and Leaves Open
The idea of using a discriminator as a pseudo-label filter, and the evidence that it can beat thresholding, both come from Adversarial Learning for Semi-supervised Semantic Segmentation. That work trains a fully convolutional discriminator on predicted versus ground-truth label maps and trusts only regions it rates as real. On 1/8 splits it reports gains from 55.5% to 58.8% mIoU on Cityscapes and from 66.0% to 69.5% on PASCAL VOC with a DeepLab-v2 network. The same work notes that its discriminator overfits by detecting the one-hot sharpness of true labels. It applies no remedy, and it uses a convolutional backbone where compute is not the bottleneck.
The remedies come from the low-data GAN literature, and we reuse them:
-
Training Generative Adversarial Networks with Limited Data (ADA) adapts augmentation strength using an overfitting heuristic computed from discriminator outputs. - Deceive D: Adaptive Pseudo Augmentation (APA) adaptively relabels generated samples as "real". On FFHQ-5k it improves FID from 37.83 to 13.25 at no extra cost.
-
Quality Aware Dynamic Discriminator Rejection Sampling (QADDRS) withholds the easiest real and fake samples from the discriminator's update.
All three operate on whole images in image synthesis. None has been applied to dense, per-patch judgments of label maps. Moving them there is part of what we build.
The exit mechanism comes from Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation (DToP). DToP attaches auxiliary heads between transformer stages and finalizes confident patches early, keeping the k most confident patches per class as context anchors. It cuts 20–35% of GFLOPs on ADE20K, reaching 49.8% mIoU against a 49.7% baseline with ViT-Base. DToP is fully supervised and used for inference only. Its exit criterion is softmax confidence, which is exactly the signal Combating Noise shows to be unreliable in an ambiguous "danger zone".
Auto-Annotation Quality Prediction shows that a learned quality filter helps most on rare classes. On Cityscapes it raises the "train" class from 55.2% to 74.9% IoU. However, it needs an 18-model ensemble, which is the opposite of saving compute.
Research Program
The segmentation model is a plain ViT-Base with DToP-style auxiliary heads after stages two and three.
The discriminator. It operates on the coarse patch grid, not on pixels, and uses windowed local attention or convolutions. Its own cost is therefore linear in the number of patches and small next to the backbone. It receives a local window of class-probability vectors and outputs a per-patch realism score $s_i \in [0,1]$. To remove the sharpness shortcut Hung et al. observed, we feed real and predicted maps in the same format. Both are temperature-softened, so the discriminator cannot win by spotting one-hot vectors.
Overfitting control. We hold out a small slice of the annotated maps. We track $$r_t = \mathbb{E}_{\text{train}}[\operatorname{sign}(D)] - \mathbb{E}_{\text{held-out}}[\operatorname{sign}(D)],$$ the gap at step $t$ between how confidently the discriminator calls annotated training maps real and how confidently it calls held-out annotated maps real. This adapts the ADA heuristic to dense outputs. When $r_t$ exceeds a target, two things happen. First, we raise the probability that high-agreement predicted patches are presented as "real", which is a dense analogue of APA. Second, we withhold the most extreme-scoring patches from the discriminator update, a per-patch version of QADDRS.
Gating and weighting. At each exit stage, every image exits a fixed fraction $K$ of its patches, chosen by top-$s_i$. A fixed per-image budget keeps tensor shapes uniform for GPU efficiency. Which Tokens to Use? supports top-K as a strong selection rule. DToP's per-class anchor rule is kept so that rare classes are not wiped out. The pseudo-label loss on unannotated images is weighted by $s_i$. Exited patches are revisited at full depth once every $M$ epochs, so a frozen error can still be corrected and we can measure how often that happens.
Baselines
Supervised-only. The naive reference is the same ViT-Base trained on the annotated split alone, at full depth. Every semi-supervised gain is measured against it.
Strongest compute-efficient competitor: self-confidence gating. This is DToP's own rule moved into semi-supervised training: exit and pseudo-label using auxiliary-head softmax confidence, with the same exit budget $K$ and anchor rule. It is the strongest competitor because it costs nothing extra and directly embodies the "just use confidence" position. It gets the same backbone, data split, schedule, and the same tuning budget, a sweep over threshold and $K$ matched in GPU-hours to ours. Beating it means higher mIoU at equal training FLOPs, or equal mIoU at clearly lower training FLOPs. In either case the exited-patch pseudo-label precision must also be higher.
Strongest existing quality-oriented method. This is Hung et al.'s adversarial semi-supervised scheme, reimplemented on the same ViT-Base backbone at full depth with no exits. The reported reference is 58.8% mIoU on Cityscapes 1/8 and 69.5% on PASCAL VOC 1/8, from Adversarial Learning for Semi-supervised Semantic Segmentation. Those numbers are for DeepLab-v2, so we re-run the method rather than compare against them. Beating it means matching its mIoU while spending 20% or more fewer training FLOPs. We also run a full-depth version of our discriminator to separate the gain from overfitting control from the gain from exiting.
Accuracy reference. We include FixMatch-style fixed-threshold pseudo-labeling without exits as an accuracy reference. No reported result for FixMatch on these splits was found on Lacuna, so it is run by us under the same budget.
Ablations. The key ablation is our full method with the overfitting controls switched off: no $r_t$-driven injection or rejection, and no format matching. Its expected failure is the central evidence for the claim. Further ablations switch off one control at a time and drop the periodic revisit.
Why not just…?
Why not just use the auxiliary heads' softmax confidence, as DToP does, for both exits and pseudo-labels? This is the real threat. If it works as well as our method, the discriminator is unnecessary. There is indirect evidence against it. Thresholded softmax is fooled in the danger zone (Combating Noise), and structure-aware discriminators beat it for pseudo-labeling (Hung et al.). But neither paper tested training-time exits, so the question is open. The confirmation-lock experiment below settles it directly. If self-confidence exits show no rising error over training, we will report the discriminator as unnecessary for gating and keep it only for pseudo-label weighting, where its value can be tested separately.
Why not just add DiffAugment or ADA to Hung's discriminator and skip the exits? That improves pseudo-label quality but saves no compute. On Cityscapes at full resolution, the full-depth transformer is the bottleneck. Also, color and cutout augmentations are poorly defined on label maps, which is why we use APA- and QADDRS-style controls instead. The full-depth variant of our discriminator, listed among the baselines, is exactly this alternative. It tells us how much accuracy the exits cost.
Why not just freeze a pretrained foundation model as the discriminator, as Ensembling Off-the-shelf Models for GAN Training does? That paper freezes pretrained vision backbones to stop memorization. Those backbones are pretrained on natural images, while our discriminator judges class-probability maps, which they have never seen. Running several full backbones would also add exactly the quadratic attention cost we are trying to remove. We include one frozen DINO-on-colorized-label-map variant as a sanity check, but we expect it to fail.
Why not just downsample, or use FlashAttention-2? Downsampling discards the small, rare objects where pseudo-labels matter most. SG-Former shows that adaptive token allocation beats uniform coarsening on dense tasks. FlashAttention-2 reduces memory traffic but leaves the quadratic compute unchanged. LLSA reports a 6.09x training speedup over it at 256×256 pixel sequences. We use FlashAttention-2 in every arm, so any saving we report comes on top of it.
Proposed Experiments
Data and metrics. We run Cityscapes at 1/8 and 1/4 annotated splits and PASCAL VOC 2012 at 1/8. We also use ADE20K with 1/8 of its labels, for which no reported semi-supervised result was found on Lacuna. These datasets are fully annotated, so the hidden true labels of the "unannotated" images let us measure pseudo-label precision exactly. We report:
- mIoU, plus IoU on rare classes such as Cityscapes "train";
- training GFLOPs and measured wall-clock time;
- the discriminator overfitting gap $r_t$;
- the precision of exited patches over training.
Confirmation-lock test. We track the fraction of exited patches whose frozen label is wrong, together with the fraction later corrected at revisit, across training for each gating rule. Our claim predicts this error rises under self-confidence gating and stays flat under the controlled discriminator.
Dose-response test. We vary discriminator capacity and annotated-set size, then plot exit precision against $r_t$. If exit precision does not fall as $r_t$ grows, overfitting is not the mechanism and the claim is wrong. If the unregulated discriminator does not end up below softmax gating, the claim that overfitting control is necessary is also wrong.
Compute test. We report wall-clock training time with batched fixed-$K$ gathering, not just FLOPs. If uniform shapes still do not produce at least a 15% wall-clock reduction on Cityscapes at full resolution, the compute half of the claim fails, and we will say so.
References
- Adversarial Learning for Semi-supervised Semantic Segmentation
- Training Generative Adversarial Networks with Limited Data
- Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
- Improving the Training of Data-Efficient GANs via Quality Aware Dynamic Discriminator Rejection Sampling
- Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation
- Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
- Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers
- Ensembling Off-the-shelf Models for GAN Training
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers