Borrowing Structure Without Borrowing Errors: Label-Free Affinity Distillation for Small Segmentation Models

Semantic segmentation means labeling every pixel of an image with a class. The labels are expensive, often more than an hour per Cityscapes street scene, so practitioners train with a few hundred labeled images and many unlabeled ones. This setting is called semi-supervised learning. The models that do best in it use very large pretrained encoders. Phones, robots and cars, however, need small networks such as ResNet-18 or MobileNetV2. The usual way to bridge the two is knowledge distillation, in which a small "student" network is trained to imitate a large "teacher." The trouble is that in the low-label regime the teacher is itself trained on scarce labels. It is confidently wrong on rare classes, thin objects and boundaries, and the student copies those mistakes faithfully.

We propose to split what the student is taught according to where each kind of knowledge comes from. The answer to "which class is this pixel?" can only come from a teacher trained on the scarce labels, so it carries that teacher's errors. The answer to "which pixels belong together?" can come from a frozen self-supervised encoder such as DINOv2, which never saw the target labels. Its errors therefore do not grow as labels shrink, and they are not reinforced by the student's own pseudo-labels. We distill this label-free grouping structure into the small student with one relational loss. Class identity continues to come from the label-trained teacher. The prediction is testable: the advantage of this split over plain teacher distillation should grow as the number of labels falls.

Core Claim

In data-scarce segmentation, the pairwise structure of a frozen self-supervised encoder is a more reliable distillation target than the pairwise structure of a teacher trained on the scarce labels. The gap widens as labels decrease. A lightweight student that takes class identity from a strong semi-supervised teacher and pixel-grouping structure from the frozen encoder should therefore beat both plain logit distillation and existing lightweight semi-supervised distillation. The gains should be largest at the smallest label budgets, on boundaries, and on rare classes.

The closest existing work falls into three groups. The first is lightweight semi-supervised distillation. Multi-granularity Distillation (MGD) trains ResNet-18 and MobileNet-v2 students from two complementary CNN teachers using image-, region- and pixel-level losses. Triple-View Knowledge Distillation distills a ConvNet teacher and a ViT teacher into a hybrid student. Mutual Knowledge Distillation cross-supervises two student/mean-teacher pairs. In all three, every teacher signal, including MGD's region self-correlation, is produced by networks trained on the same scarce labels, so the structural signal inherits the labeling errors. This is the gap we target.

The second group is dense distillation under full supervision. Structured Knowledge Distillation introduced pair-wise affinity distillation for segmentation. Boundary and Relation Distillation adds boundary and self-relation terms. We reuse their affinity-matching mechanism, but move the source of the affinities from a supervised teacher to a label-free encoder and move the setting to semi-supervised learning.

The third group uses foundation models in semi-supervised learning. ConformalSAM shows that raw foundation-model pseudo-labels perform worse than labeled-only training unless they are filtered. It uses foundation models only as hard-mask generators and then drops them in a second stage. UniMatch V2 shows that DINOv2 encoders dominate older backbones, but it never compresses them. ConformalSAM constrains our design: it warns against trusting foundation-model class predictions on the target domain, which is exactly why we take only class-agnostic grouping from the frozen encoder.

DAFormer is the nearest in spirit. It regularizes an adapted model toward its own ImageNet features for object classes. Ours differs in three ways: it is a cross-architecture compression setting, the target is relational rather than a raw feature distance, and the encoder is self-supervised rather than ImageNet-supervised.

Research Program

The system has three parts. The class teacher is UniMatch V2 with a DINOv2-B encoder, trained on the labeled split and the unlabeled pool. The structure source is a frozen DINOv2-B encoder that receives no target labels. Following UniMatch V2's finding that freezing cuts memory sharply, we run it without gradients on the weakly augmented view. The student is DeepLabV3+ on ResNet-18 or MobileNetV2.

The student trains on three signals:

  1. cross-entropy on labeled images;
  2. a temperature-softened logit distillation loss toward the class teacher on all images;
  3. one affinity loss.

For each position $i$ on a downsampled feature grid, let $f_i$ be the frozen encoder feature and $s_i$ the student feature after a small projection head. Let $N(i)$ be a neighborhood made of a local window plus a few randomly sampled distant positions. We turn similarities into a distribution over neighbors,

$$P^{F}_{ij} = \frac{\exp(\cos(f_i, f_j)/\tau)}{\sum_{k \in N(i)} \exp(\cos(f_i, f_k)/\tau)},$$

and define $P^{S}_{ij}$ the same way from student features. Here $\tau$ is a temperature. The affinity loss is the average over $i$ of $\mathrm{KL}(P^{F}_{i} \,\|\, P^{S}_{i})$.

We deliberately add only this single structural term. Relative Difficulty Distillation reports gradient conflicts when many dense distillation losses are stacked. The sparse neighborhood keeps memory linear in the number of pixels rather than quadratic.

The analysis component matters as much as the method. Before any student is trained, we measure affinity quality directly on held-out ground truth. For pixel pairs, we ask how well each source's similarity ranks same-class pairs above different-class pairs. We compare three sources across label budgets:

  1. the frozen encoder;
  2. the semi-supervised class teacher's features;
  3. MGD-style CNN teachers.

The frozen encoder's curve is flat by construction. The open question is whether, and at what budget, it lies above the label-trained curves.

Baselines

The naive reference is the student trained on the labeled images alone. A second weak reference is the student trained with plain mean-teacher self-training. MGD reports that mean teacher and Cross Pseudo Supervision degrade badly on ResNet-18, so this tells us how much any distillation helps.

The strongest competitor is a two-stage pipeline. First train the UniMatch V2 DINOv2-B teacher on the labeled split and the unlabeled pool. Then distill its soft logits on every image into the same student, with the same schedule and augmentations as our method. This baseline uses the best available teacher, needs no new machinery, and is what a practitioner would try first. Our method is this baseline plus the affinity term, so the comparison isolates the new ingredient exactly. UniMatch V2 reports a +6.9% mIoU gain of DINOv2-S over the ResNet-101 state of the art on PASCAL VOC. We found no absolute mIoU on Lacuna for a distilled lightweight student of UniMatch V2, so we will establish this number ourselves.

The strongest published lightweight semi-supervised method is MGD. It reports 66.86% mIoU for a ResNet-18 student on PASCAL VOC 2012 and an 11.5% gain for ResNet-18 at the Cityscapes 1/16 partition. We will rerun it on our exact splits rather than rely on reported numbers.

Mutual Knowledge Distillation reports 65.35% mIoU with 92 PASCAL VOC labels, using a larger backbone. It serves as a reference point for what heavy semi-supervised distillation achieves.

We also report fully supervised distillation upper bounds on Cityscapes with all labels:

  1. 77.03% mIoU for a ResNet-18 student under Af-DCD;
  2. 77.18% for a ResNet-18 student under RDD, whose teacher scores 78.07%.

These bound how close a semi-supervised student can get.

Two ablations isolate the key ingredient:

  1. Keep everything fixed but compute the affinities from the class teacher's own features instead of the frozen encoder. This tests whether label-free provenance matters, not merely the presence of a relational loss.
  2. Replace the frozen DINOv2 with an ImageNet-supervised encoder. This separates "self-supervised" from "simply pretrained and frozen."

Fairness rules apply to every arm. All methods share the student architecture, splits, crop size, iteration count and augmentation, and each gets the same small search over its loss weight on a held-out slice of the labeled data. We report GPU-hours and peak memory alongside accuracy.

Beating the two-stage baseline means a consistent mIoU gain across three seeds at the 92- and 183-label PASCAL VOC budgets and the Cityscapes 1/16 budget. The gain must exceed seed variance, and it must be larger at those budgets than at the 1/2 budget.

Why not just…?

Why not just distill the strongest semi-supervised teacher's logits and stop there? This is the main threat. Logits already carry inter-class similarity, and a DINOv2-based teacher's predictions implicitly encode its encoder's grouping. If the teacher-feature affinity ablation does as well as the frozen-encoder version, or if the two-stage baseline matches our method at every budget, the core claim fails. The affinity-quality diagnostic predicts which outcome to expect before training any student: frozen-encoder affinities must be measurably cleaner than teacher affinities at low budgets, or there is nothing to transfer.

Why not just deploy DINOv2-S with a frozen encoder, as in UniMatch V2? It is already 24.8M parameters and strong. However, a ViT's attention cost grows quadratically with token count at Cityscapes resolution, which conflicts with the edge budgets that motivate MGD's 59.6× FLOP compression. We will measure latency on a fixed device for DINOv2-S at reduced resolution and for our student. If DINOv2-S matches both speed and accuracy, we will report that the compression problem is less pressing than assumed.

Why not just use the frozen encoder's affinities to clean the pseudo-labels, then distill normally? This alternative propagates labels along DINOv2 similarity before distillation, much as ConformalSAM filters hard masks. It is a credible competitor and may capture most of the gain. We run it as a full arm. If it matches the loss-based version, the contribution narrows to the finding that label-free structure is the right correction signal, and we will say so. The key difference is that refinement still commits to hard class decisions. The affinity loss transfers grouping even where the class label stays uncertain, which matters most at boundaries.

Why not just use MGD's region self-correlation loss? MGD already distills relational structure, but from label-trained CNN teachers. Running MGD's loss with the frozen encoder as its source, against its original teachers, directly tests whether the source or the loss form drives the result.

Proposed Experiments

The benchmarks are three. On PASCAL VOC 2012 we use 92, 183, 366, 732 and 1464 labels, with the augmented set as the unlabeled pool. On Cityscapes we use the 1/16, 1/8, 1/4 and 1/2 partitions. ADE20K serves as a many-class stress test. Metrics are:

  1. mIoU;
  2. mIoU on the rarest third of classes;
  3. a boundary-band mIoU computed within a few pixels of ground-truth edges, where Boundary and Relation Distillation locates the failure modes of lightweight students;
  4. training memory and time.

The first decisive check is the affinity-quality diagnostic described above. If frozen-encoder affinities are not better than teacher affinities at low budgets, we stop.

The second is the shape of the gain curve. Our method's improvement over the two-stage baseline should shrink monotonically as labels increase. A flat curve would mean the affinity term is a generic regularizer rather than a fix for scarcity.

The third is localization. The gain should be concentrated in boundary bands and rare classes, with the class teacher's per-class errors predicting where the student improves.

The fourth is the provenance ablation, frozen-encoder versus teacher-feature affinities, at fixed loss form. Together these four checks determine whether the benefit comes from structure that never saw the scarce labels or merely from adding one more distillation term.