Small segmentation networks, the kind that run on phones and cars, blur object edges and break thin structures. Knowledge distillation trains a small "student" to copy a large "teacher." Applied to segmentation, it usually compares the two models pixel by pixel. That comparison treats a wrong pixel in the middle of a road the same as a wrong pixel on a pedestrian's outline, even though only the second one changes the picture. A separate line of work, reconstruction-based regularization, makes a segmentation network redraw the input image from its own predicted label map. A sloppy label map cannot explain where the image's edges are, so this redrawing step sharpens boundaries without needing boundary annotations.
We propose to combine the two in a specific way. First, train a small decoder that turns the teacher's soft label maps back into images, and freeze it. Then, during distillation, pass the student's label map through that frozen decoder and require the result to match the decoder's output for the teacher's label map. The target is not the raw image. It is the part of the image that the teacher's layout can explain. The student is penalized for layout errors that would visibly change the scene, which in practice means errors on boundaries and thin structures. It is not asked to encode texture, lighting, or clutter that no label map could account for. The loss needs no ground-truth edges, so it works on unlabeled images. It acts only on output label maps, so it does not care whether teacher and student are CNNs or transformers.
Core Claim
Let $p_t(x)$ and $p_s(x)$ be the teacher's and student's per-pixel class-probability maps for image $x$. Let $D$ be a lightweight convolutional decoder from probability maps to RGB images. $D$ is trained once on the teacher's outputs to minimize $\|D(p_t(x)) - x\|_1$, then frozen. The proposed layout-reconstruction distillation loss is
$$ \mathcal{L}_{LR}(x) = \big\| D(p_s(x)) - D(p_t(x)) \big\|_1 , $$
which is added to the usual pixel-wise KL distillation loss and, when labels exist, to cross-entropy. Since $D$ is frozen, gradients reach only the student.
Through $D$, a change in the student's label map costs roughly as much as the change it causes in the image the teacher's labels explain. On a large uniform region, a small probability shift barely moves $D$'s output. On an edge where a "pole" pixel turns into "sky," the redrawn image changes sharply. The loss therefore works as a learned, image-aware metric between label maps.
The claim we will test is narrow. A teacher-anchored, frozen decoder gives better boundaries and better overall accuracy than three alternatives: a jointly trained reconstruction branch aimed at raw RGB, simple image-gradient weighting of the KD loss, and feature-space generative distillation. Each comparison is made at matched training cost.
Relation to Prior Work
The idea that reconstruction regularizes boundaries comes from Deep Dual Learning for Semantic Image Segmentation. That work predicts a label map, reconstructs the RGB image from it with a jointly trained decoder, and reports 86.8% mIoU on PASCAL VOC with far fewer full annotations. Image-level to Pixel-wise Labeling: From Theory to Practice uses similar label-to-image generation. Neither involves a teacher. Both reconstruct the raw image, so they inherit the tension the research file flags: matching raw RGB pushes the network to retain task-irrelevant texture. A compact student has the least spare capacity for that. Our frozen, teacher-conditioned target is designed to remove that texture burden, and we test whether it does.
Two existing methods already pair reconstruction with distillation, but in a different place. Masked Generative Distillation (MGD) masks student features and regenerates the teacher's feature map. It reconstructs in latent space, with no pixel-level or boundary anchor. Distillation Learning Guided by Image Reconstruction for One-Shot Medical Image Segmentation trains a U-Net teacher to reconstruct real images from its encoder features and transfers that ability to a student, in an atlas-registration medical setting. In both, reconstruction is conditioned on encoder features, not on the predicted label map, so neither forces the segmentation output itself to account for image structure. Masked Autoencoders Enable Efficient Knowledge Distillers mixes masked pixel reconstruction with feature distillation, but only for ImageNet pre-training. It shows reconstruction and distillation are compatible, and says nothing about dense outputs.
The boundary-focused distillation methods supply the problem statement and the main competitors. Structured Knowledge Distillation for Semantic Segmentation adds pairwise affinity distillation and a discriminator on predicted maps. BPKD splits distillation into edge and body regions using ground-truth edge masks with a hand-set width. Boundary and Relation Distillation for Semantic Segmentation derives boundaries from feature affinities and adds a full self-relation matrix. Cross-Image Relational Knowledge Distillation extends relations across images with memory banks. These methods either need ground-truth edges, train an adversary, or pay quadratic cost in pixel count. Our loss is linear in pixels, uses no edge labels, and has a deterministic gradient.
Cumulative Spatial Knowledge Distillation for Vision Transformers constrains our design in one respect. It found that a fixed local spatial constraint can cap a student's capacity, and it decays that constraint during training. We therefore treat a decay schedule for $\mathcal{L}_{LR}$ as an ablation, not an assumption.
Research Program
The work has three parts.
The first part builds and characterizes the frozen decoder. $D$ is a small convolutional upsampling network, kept deliberately weak, with a few layers and no skip connections from the image. A weak decoder can only redraw what the layout encodes. We train it on teacher soft maps over the training images, in two variants: unlabeled images only, and labeled images only. We measure its redrawing quality. We also visualize the per-pixel sensitivity $\|\partial D / \partial p\|$ to confirm it peaks at class transitions. If it does not, the mechanism is not doing what we claim.
The second part trains students with $\mathcal{L}_{LR}$ on fully labeled data. It then adds the 20,000 Cityscapes coarse images as unlabeled data, using only teacher predictions and ignoring the coarse labels. Earlier results motivate this. Structured KD reports a +13.5% mIoU gain on CamVid from soft targets on unlabeled data, and the dual-learning work shows reconstruction can supervise boundaries without boundary labels. We expect $\mathcal{L}_{LR}$ to transfer more boundary information per unlabeled image than pixel KL does.
The third part tests the architecture-agnostic claim. We distill a CNN teacher into a small ViT-based segmentation student, where feature-matching methods need projection heads. $\mathcal{L}_{LR}$ needs nothing extra there.
Baselines
The naive reference is the student trained with cross-entropy only, the "do nothing" point. On Cityscapes with DeepLabV3-ResNet18, MGD reports 73.20 mIoU and CIRKD reports 74.21. We will re-run this baseline ourselves rather than choose between those numbers. A second reference is the student with pixel-wise KL distillation alone, which isolates what $\mathcal{L}_{LR}$ adds on top of standard KD.
The strongest existing competitors are MGD and CIRKD. MGD is the closest reconstruction-plus-distillation method and reports DeepLabV3-ResNet18 improving from 73.20 to 76.02 mIoU on Cityscapes (Masked Generative Distillation). CIRKD reports 76.38 mIoU for the same student on Cityscapes (Cross-Image Relational Knowledge Distillation), the highest absolute student number in the file. BPKD targets boundaries most directly but reports its results as relative gains, so we will re-run it rather than quote a figure. We will use the same teacher, a DeepLabV3-ResNet101 trained once and shared across all methods, along with the same student initialization, crop size, schedule, and a comparable small hyperparameter search for every method. Beating the competition means exceeding re-run MGD and CIRKD on Cityscapes mIoU and clearly exceeding them on boundary-band (trimap) mIoU. We will also report our re-run numbers next to the published ones, so readers can see any gap in our reproduction.
Three ablations isolate the key ingredient. The first, "DIS + KD," trains a decoder jointly with the student to reconstruct the raw RGB image from the student's label map, with pixel KL added. This is the obvious additive combination. The second, "Sobel-weighted KD," weights pixel KL by the image-gradient magnitude, with no decoder. The third uses a frozen decoder with random weights, which tests whether the teacher-learned image explanation matters at all. Secondary ablations cover a decoder trained on ground-truth labels instead of teacher soft maps, decoder capacity, and a decaying versus constant weight on $\mathcal{L}_{LR}$.
Why not just…?
Why not just add the dual-learning reconstruction loss to ordinary KD? This is the most natural combination and the main threat. Its target is the raw image, so the student must try to make its label map account for texture and lighting that no label map can explain. The resulting gradients are noisy, which is the reconstruction-versus-invariance tension the file identifies. A jointly trained decoder can also learn to hallucinate structure, which weakens the pressure on the student. The DIS+KD ablation, run at matched training compute, settles the question. If it matches $\mathcal{L}_{LR}$, our contribution reduces to "dual learning plus KD works." We would then report that result honestly as the main finding.
Why not just weight the KD loss by image edges (Sobel) or ground-truth boundaries, as in BPKD? Image-gradient weighting is cheap and may capture much of the benefit. That is exactly why it is an ablation. It differs from our loss in two ways. It fires on every texture edge, including shadows and road markings, rather than only on edges the teacher's layout explains. And it reweights a per-pixel loss instead of scoring the joint configuration of a neighbourhood. Ground-truth boundary masks, as in BPKD, cannot be used on unlabeled coarse images, and they need a hand-tuned width. If Sobel-weighted KD matches us on labeled Cityscapes, the remaining argument rests on the unlabeled-data experiment. If it matches us there too, the decoder is not worth its cost.
Why not just use MGD-style feature regeneration? MGD already shows generative recovery beats plain feature mimicry. But it reconstructs teacher features, which say nothing direct about whether the predicted label map is geometrically right. It also requires compatible feature maps between teacher and student. The head-to-head on boundary metrics answers the first point. The CNN-to-ViT experiment, where MGD needs projection heads and our loss does not, answers the second.
Proposed Experiments
The main benchmark is Cityscapes (The Cityscapes Dataset), using fine labels for supervised training and the coarse split as unlabeled data. We report mIoU, trimap mIoU in bands of several widths around ground-truth boundaries, boundary F-score, and per-class IoU on thin classes such as poles and traffic signs. ADE20K (Scene Parsing through ADE20K) tests whether the decoder still helps with 150 classes, where redrawing images from label maps is much harder. PASCAL VOC 2012 is used for comparison with the dual-learning and boundary-relation results. Students are DeepLabV3-ResNet18 and a MobileNetV2-based model. The cross-architecture study adds a small ViT student.
Five checks are decisive. First, $\mathcal{L}_{LR}$ must beat DIS+KD and Sobel-weighted KD at matched compute. Without that, the teacher-anchored frozen target adds nothing. Second, the gain over pixel KD must be concentrated in the narrow trimap bands. If it is spread uniformly, the boundary story is wrong. Third, the random frozen decoder must give roughly the KD-only result, showing that what $D$ learned is the active ingredient. Fourth, adding unlabeled coarse images must widen our margin over pixel KD, because that is where a label-free boundary signal should pay off. Fifth, decoder capacity is a control. If a high-capacity decoder helps more than a weak one, the student may be getting texture-reconstruction pressure through the back door, and the claimed separation between layout and texture would not hold.
References
- Deep Dual Learning for Semantic Image Segmentation
- Image-level to Pixel-wise Labeling: From Theory to Practice
- Masked Generative Distillation
- Distillation Learning Guided by Image Reconstruction for One-Shot Medical Image Segmentation
- Masked Autoencoders Enable Efficient Knowledge Distillers
- Structured Knowledge Distillation for Semantic Segmentation
- BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation
- Towards Customized Knowledge Distillation for Efficient Dense Image Predictions
- Cross-Image Relational Knowledge Distillation for Semantic Segmentation
- Cumulative Spatial Knowledge Distillation for Vision Transformers
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Scene Parsing through ADE20K Dataset