An ensemble of segmentation networks usually beats any single member, but running several networks per frame costs roughly as many times the latency and memory. The standard fix is distillation: train one small "student" network to copy the ensemble. For dense prediction, copying only the final per-pixel outputs throws away much of what the ensemble knows, so feature-level distillation matches the student's internal feature maps to each teacher's. We point out a problem that this feature matching ignores. Networks with different backbones, strides and upsampling paths place the same object boundary at slightly different spatial positions in their feature maps; FaPN and SFNet showed that such shifts exist even inside a single network. A rigid feature-matching loss therefore asks the student to copy each teacher's geometric quirks along with its semantics. When the teachers disagree about geometry, those demands conflict.
We propose to place a small, training-only "warp-only bridge" between the student and each teacher. The bridge predicts a smooth displacement field, called a flow, that slides the student's features into each teacher's coordinate frame before the loss is computed. A per-pixel gate decides which aligned teacher the student should trust at each location, in the style of a mixture of experts: a set of specialist sub-networks plus a small network that weights them per input. Both the bridges and the gate are thrown away after training. The deployed student therefore costs exactly one small network, and the ensemble's alignment machinery and expert routing are paid for only during training.
Core Claim
Feature-level ensemble distillation for dense prediction is limited by geometric disagreement between teachers, not only by student capacity. If the bridges are restricted so they can move features but not rewrite them, the student can absorb each teacher's semantics without absorbing its spatial artifacts. This should produce a better single-model student than rigid bridges at identical inference cost. The restriction matters. A bridge that can both warp and freely transform features could absorb the whole discrepancy and leave the student with no training signal. The proposal therefore makes a specific architectural bet: the bridge's non-geometric capacity is capped at a per-channel 1×1 projection, and all spatial freedom goes into a smoothness-regularized flow.
Relation to Prior Work
The closest work is FEED. Its parallel variant trains one nonlinear transformation layer per teacher, each a stack of three rigid 3×3 convolutions, and discards them at test time. On CIFAR-100 with WideResNet 28-10 this gives 16.86% error, against 18.30% for logit distillation. We reuse its discard-after-training structure but change what the bridge is allowed to do. FEED's bridges are spatially rigid and unconstrained in content; ours are spatially flexible and constrained in content. FEED was also evaluated on classification, where only pooled features matter, whereas we target dense prediction, where geometry matters.
Adaptive Knowledge Amalgamation uses 1×1 "transfer bridges" that match channel dimensions across heterogeneous teachers, plus an entropy-based selector. It also cannot correct spatial shifts, and its selection works per sample rather than per pixel.
The alignment operator itself comes from SFNet's Flow Alignment Module, which predicts a dense 2D offset field with a few 3×3 convolutions and warps features by bilinear sampling. SFNet uses this inside the network at inference. We use it only as a training-time loss adapter, so it adds no irregular memory access at deployment. This matters because Algorithm-hardware Co-design for Deformable Convolution documents that fractional-coordinate sampling runs slower than its FLOP count suggests.
The per-pixel teacher gate is motivated by Spatial Mixture-of-Experts and the pixel-level experts of X-MoGe. Both keep their routing at inference, and SMoE reports slower wall-clock time than dense CNNs as a result. We keep routing only in the loss.
One prior result directly constrains the gate design. Rethinking Ensemble-Distillation for Semantic Segmentation Based UDA found that per-class channel-wise selection of teachers beat per-pixel fusion by 4.72% mIoU on GTA5→Cityscapes. We treat this as a hypothesis to test rather than ignore. Their pixel-wise fusion combined unaligned teacher predictions, so our hypothesis is that per-pixel selection failed partly because neighbouring pixels drew from teachers whose boundaries disagreed. If per-pixel gating still loses to channel-wise selection after alignment, we adopt channel-wise selection. The alignment claim does not depend on this choice.
Research Program
Let $s$ be the student's feature map at a chosen pyramid level and $t_k$ the feature map of teacher $k$, for $k = 1, \dots, K$. Bridge $k$ predicts a flow field $u_k$ from the concatenation of $s$ and $t_k$, warps the student by bilinear sampling to get $\mathcal{W}(s, u_k)$, and applies a 1×1 channel projection $P_k$. A gate $g_k(x)$ gives per-pixel weights that sum to one over teachers at each pixel $x$. The training loss added to the ordinary segmentation loss is
$$\mathcal{L} = \sum_x \sum_{k=1}^{K} g_k(x)\,\big\| P_k \mathcal{W}(s,u_k)(x) - t_k(x) \big\|^2 + \lambda \sum_k \|\nabla u_k\|^2 + \beta\,\mathcal{H}(g),$$
where the second term penalizes rough flows with weight $\lambda$ and the third is a small entropy term with weight $\beta$ that discourages the gate from settling early on one teacher.
The gate takes each teacher's per-pixel prediction confidence and the post-warp residual as inputs. To avoid the collapse behaviour that MoE routers are known for, it is initialized uniform. As an ablation, the gate's input is restricted to confidence alone.
The flow predictor is run coarse to fine across pyramid levels. This addresses the limitation that a single 3×3 flow predictor cannot capture displacements larger than its receptive field.
The warp is detached from the segmentation head. The student's own predictions always come from unwarped features, so the student is trained to be right in image coordinates.
Baselines
The naive reference points are the student trained alone with cross-entropy and the student with standard logit distillation from the averaged ensemble output. The upper reference is the full ensemble itself, reported together with its latency and memory.
The strongest existing method on this question is FEED's parallel variant, reimplemented for dense prediction with one three-layer 3×3 convolution bridge per teacher at the same pyramid levels. Its reported numbers are for classification only (16.86% CIFAR-100 error, 25.27% ImageNet top-1 error for ResNet-34). No reported result was found on Lacuna for FEED-style distillation on Cityscapes or ADE20K, so we run it ourselves. The second competitor is the channel-wise certainty-selected ensemble distillation of the UDA paper, run in our supervised setting and also in its native GTA5→Cityscapes setting. That paper reports relative margins, up to 6.64% mIoU over EnD, but the file gives no absolute mIoU.
Two architectural references bound what the student could gain without distillation: SFNet, whose ResNet-18 model reports 80.4% mIoU on Cityscapes at 26 FPS, and FaPN with a ResNet-18 backbone.
Fairness rules apply to every method. All methods use the same student, the same teachers, the same training schedule and the same hyperparameter search budget (an equal number of trials per method over its loss weights). Rigid bridges are given at least as many parameters as warp-only bridges. Beating FEED means a statistically reliable mIoU gain over three seeds at identical deployed FLOPs and FPS. We would consider roughly one point on Cityscapes meaningful given FaPN's reported 1.2–2.6-point gains from alignment.
Three ablations isolate the key ingredient. The first swaps the flow for a deformable-convolution bridge. The second removes the 1×1 content cap by giving the bridge 3×3 convolutions plus a flow. The third replaces the per-pixel gate with uniform averaging and with channel-wise selection.
Why not just…?
Why not just distill the ensemble's output logits? Output maps already live in image coordinates, so misalignment does not arise there. This is the simplest real threat. FEED's CIFAR-100 result (18.30% error for logit distillation against 16.86% for feature distillation) suggests features carry extra signal, but that result is from classification. The main tables report logit distillation alongside ours. If it matches warp-only feature distillation on Cityscapes, the claim fails.
Why not just give FEED's rigid bridges more capacity or a larger receptive field? A deep enough convolution stack can in principle learn a shift. However, an unconstrained bridge can also learn to absorb the whole student–teacher gap, which removes the training signal. The capacity-matched rigid baseline and the "flow plus unconstrained content" ablation test this directly. If the enlarged rigid bridge closes the gap, the geometric-disagreement hypothesis is wrong.
Why not just average the teachers' weights, SWA-style, or build the student with an alignment neck like SFNet? Weight averaging works along one training trajectory, as SWA shows. WEMoE, however, reports that static task-arithmetic merging falls to 69.0% average accuracy on its eight-dataset CLIP benchmark, and heterogeneous teachers cannot be averaged at all. We still include a same-architecture soup as a baseline. An alignment neck in the student is complementary rather than competing: we test our method on both a plain-FPN student and an SFNet-R18 student. A gain on top of SFNet would show that aligning the loss differs from aligning the architecture.
Proposed Experiments
The main setting is supervised Cityscapes, with ADE20K as a harder 150-class check. The ensemble contains three deliberately geometrically diverse teachers: a plain FPN on ResNet-101, SFNet on ResNet-50, and FaPN on ResNet-50. The students are ResNet-18 plain-FPN and SFNet models. We report mIoU, boundary-region IoU in a thin band around ground-truth edges (where we predict the gain concentrates), and deployed FPS and memory, which should be identical to the baseline student. The secondary setting is GTA5→Cityscapes, to compare directly with channel-wise ensemble distillation.
Three checks are decisive. The first is a controlled-shift test. Teacher features are displaced by known synthetic offsets that differ per teacher. Rigid-bridge distillation should degrade as the offsets grow, while warp-only bridges should stay flat, and the recovered flows should match the injected offsets. The second is a dose-response test. Across ensembles of increasing geometric diversity (three same-architecture seeds, then the mixed set above), our gain over FEED should grow with the measured mean flow magnitude between teachers. If the gain appears even among same-architecture seeds with near-zero flow, the mechanism is something other than alignment. The third is the gate question: whether per-pixel gating of aligned teachers overturns the UDA paper's channel-over-pixel result. Either outcome is informative for how future ensemble distillation should select teachers.
References
- FaPN: Feature-aligned Pyramid Network for Dense Image Prediction
- Semantic Flow for Fast and Accurate Scene Parsing
- FEED: Feature-level Ensemble for Knowledge Distillation
- Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
- Algorithm-hardware Co-design for Deformable Convolution
- Spatial Mixture-of-Experts
- X-MoGe: A Cross-Modal Adaptation Framework with Mixture-of-Experts and Geometry Guidance for Heterogeneous Collaborative Perception
- Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaption
- Averaging Weights Leads to Wider Optima and Better Generalization
- Merging Multi-Task Models via Weight-Ensembling Mixture of Experts