Suppose a small segmentation network is trained to copy an ensemble of large, architecturally different teachers, for example a convolutional network and a vision transformer. The teachers will disagree, and they tend not to disagree at random. Our hypothesis is that they mostly confuse classes that are semantically close, such as wall versus building or car versus truck, while agreeing on the coarser region those classes jointly occupy. Current practice discards this structure. The usual choices are to average the teachers' probabilities into one blurry target, or to pick one teacher per class channel. Either way, the student is then trained with a pixel loss or a per-class overlap loss that treats every class as independent.
We propose training the student to be the median of its teachers under a true overlap metric: the Jaccard metric loss, a soft-label-valid version of intersection-over-union. The loss is applied both to individual classes and to a small learned set of class groups. At the group level the teachers agree, so the student receives a sharp region-overlap signal there. At the class level it receives an honestly soft signal where the teachers genuinely conflict. The group layer costs $O(L\cdot d)$ per pixel for $L$ classes and $d \ll L$ groups, so it grows linearly with the label vocabulary rather than quadratically.
Core Claim
The claim has two parts. First, when teachers come from different model families, measuring the student against each teacher separately with a metric region loss beats measuring it against the teachers' average, especially when one teacher is overconfident or weak. The per-teacher sum pulls the student toward a region-level median rather than a mean. Second, adding a region loss on soft unions of correlated classes transfers inter-class structure that channel-wise fusion throws away, and does so without building a pairwise label graph.
Both parts depend on a checkable empirical premise: heterogeneous teachers' disagreements concentrate inside groups of related classes. If that premise fails, the group term has nothing to exploit, and we will report that outcome.
Where this sits
Two papers supply the loss. Jaccard Metric Losses builds Jaccard losses that are true metrics on $[0,1]^p$, so they remain correct on soft targets. It uses the triangle inequality to bound a distilled student's error, and distills from a single teacher with class channels summed independently. Dice Semimetric Losses does the analogous thing for Dice, but only as a semimetric, and shows that soft-label training cuts calibration error on QUBIQ Prostate from 0.51 to 0.20. We reuse the Jaccard metric rather than the Dice semimetric specifically because we need the exact triangle inequality. Neither paper considers multiple teachers or any structure across labels.
Two papers motivate the ensemble side. Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaption shows that naive pixel averaging of diverse teachers lets overconfident members dominate. It fixes this with output unification plus per-class teacher selection, which beats pixel-wise fusion by 4.72 mIoU on GTA5→Cityscapes. That selection treats class channels as independent, and this is exactly the structure we try to keep. Transformer-CNN Cohort shows that pairing a transformer with a convolutional network avoids the mutual confirmation of errors seen in identical-architecture pairs. It does not compress the pair into a small deployable model. Self-Decoupling and Ensemble Distillation groups classes across internal expert decoders by pixel frequency. It shares one backbone, so it has no real architectural diversity, and its grouping ignores semantics. Our groups are instead learned from where the teachers disagree.
Three papers constrain the design on the label side. ML-GCN and SigRL show that label co-occurrence helps (83.0 vs 77.1 mAP on COCO for ML-GCN), but both use dense label graphs of size $L^2$. Multi-Label Knowledge Distillation distills label-wise structure only at the image level. We take the lesson that label structure matters, and implement it as a linear pooling of class masks inside the loss rather than as a graph network.
Research Program
Let $P_S, P_{T_k} \in [0,1]^{L\times HW}$ be the per-pixel class probability maps of the student and of teacher $k$, for $k = 1,\dots,K$. Let $W \in [0,1]^{d\times L}$ be a soft assignment of classes to groups whose columns sum to one, so that $WP$ is again a valid per-pixel probability map over $d$ groups. Let $\Delta$ be the Jaccard metric loss summed over channels, and let $Y$ be the one-hot ground truth. The distillation objective is
$$\mathcal{L} = \mathrm{CE}(P_S, Y) + \frac{1}{K}\sum_{k=1}^{K}\Big[\lambda_c\,\Delta(P_S, P_{T_k}) + \lambda_g\,\Delta(WP_S, WP_{T_k})\Big].$$
Here $\lambda_c$ and $\lambda_g$ weight the class-level and group-level terms, and the cross-entropy term (the pixel-wise distribution part of the hybrid loss) supplies stable gradients early in training.
The group term preserves the metric property. $\Delta$ applied to the pooled maps $WP$ is a metric on the pooled space, so for every teacher $\Delta(WP_S, WY) \le \Delta(WP_S, WP_{T_k}) + \Delta(WP_{T_k}, WY)$. Averaging this inequality over teachers bounds the student's group-level error by its mean distance to the teachers plus the teachers' mean error.
We do not claim this bound favors per-teacher matching over mean-target matching; the analogous bound holds for the averaged teacher too. The case for the per-teacher sum is robustness, because a sum of metric distances is minimized near a median. That case must be shown empirically.
The grouping $W$ is computed once, offline, from the teacher ensemble on the training set. We accumulate, for each pixel where the teachers' top classes differ, the soft co-activation of the two classes involved. We then factor these statistics with nonnegative matrix factorization into $d$ groups and normalize the columns.
The $L\times L$ statistics matrix is only built offline. At 150 classes it is trivial, and for much larger vocabularies it can be replaced by a streaming low-rank sketch. The training-time cost of the group term is one $d\times L$ matrix multiply per pixel.
The teachers are a strong convolutional segmenter and SegFormer-B5, the transformer that Jaccard Metric Losses trains to 50.22 ADE20K mIoU. The students are PSPNet-R18 and a MobileNetV2 variant, matching the students used in the channel-wise and structured distillation literature.
Baselines
The naive reference points are the student trained with cross-entropy on ground truth alone, and the same student with cross-entropy plus a hard-label Jaccard metric loss. These measure how much any distillation adds.
The strongest single-teacher competitor is JML-KD from Jaccard Metric Losses, distilling from whichever teacher is best. We will use its confidence filtering and boundary label smoothing. No reported student result for JML-KD was found on Lacuna, so we re-run it. The closest published reference for a PSPNet-R18 student on Cityscapes is Channel-wise Distillation at 74.87 mIoU. Because that number depends on its own teacher, we re-run channel-wise distillation with our teachers rather than compare against the reported figure.
The strongest ensemble competitor is output unification plus per-class teacher selection from Rethinking Ensemble-Distillation, transplanted from domain adaptation to supervised distillation. That paper reports only relative gains (4.72 mIoU over pixel fusion), so we have no absolute number to compare against. We also include an ensemble-mean target trained with JML-KD, which is the most direct threat to our first claim.
Every method uses identical teachers, student, schedule, crop size, per-GPU batch size (which matters for region losses, as the Jaccard paper notes), and the same hyperparameter search budget of equal trials per method.
The ablations each isolate one ingredient. The first removes the group term. The second replaces learned $W$ with a random assignment of the same size. The third replaces it with ADE20K's fixed category hierarchy. The fourth uses a full $L\times L$ pairwise class-correlation distillation term, which is affordable at 150 classes. The fifth swaps the heterogeneous ensemble for an ensemble of seed-varied copies of one architecture.
We count the proposal as winning only if the full method beats both JML-KD from the best teacher and the ensemble-mean JML-KD by more than seed-to-seed variation. We will run three seeds on each benchmark.
Why not just…?
Why not just distill the average of the teachers with the Jaccard metric loss? This is the real threat. Once the loss is soft-label-correct, averaging may already absorb most of the benefit. The published evidence against averaging (Rethinking Ensemble-Distillation) concerns cross-entropy targets under domain shift, not metric region losses. We therefore run the comparison directly. We also stress-test it by adding a deliberately overconfident, weaker third teacher, where median-style matching should degrade much less than a mean target. If the two stay tied under that perturbation, we drop the per-teacher sum and keep only the group term.
Why not just distill from the single best teacher? A strong transformer may dominate the ensemble. If so, the extra teacher only adds noise, consistent with the diversity ceiling reported in Ensembling Tabular Foundation Models for similar models. The homogeneous-versus-heterogeneous ablation, together with the single-best-teacher baseline, will tell us whether the convolutional teacher's disagreements carry useful information at all.
Why not just use a full pairwise label-correlation loss, since 150 classes is small? At ADE20K scale this is fair, and we include it as an ablation. If it matches the low-rank group term, the honest conclusion is that the gain comes from structure, not from any particular parameterization. The $O(L\cdot d)$ form then matters only for larger vocabularies, which we can argue for but not test on the benchmarks in hand.
Why not use fixed super-categories instead of learned groups? A hand-made hierarchy is simpler and interpretable. Learned groups are justified only if teacher disagreement follows the teachers' failure structure rather than the ontology. The hierarchy ablation settles this.
Proposed Experiments
The first experiment is a cheap diagnostic that decides whether the rest is worth running. On ADE20K validation, we measure what fraction of pixels where the teachers disagree have their two competing classes in the same learned group, and compare against random groupings of equal sizes. We also check whether the group-level teacher agreement rate is substantially higher than the class-level rate. If disagreement is not concentrated within groups, the group term is unmotivated, and the project reduces to a pure study of median versus mean distillation.
The main experiments cover Cityscapes (19 classes) and ADE20K (150 classes) with both students. We report mIoU, per-class IoU split into frequency terciles to catch effects on rare classes, and expected calibration error. The last metric matters because soft-label metric losses were introduced partly to fix overconfidence, as shown in Dice Semimetric Losses.
The decisive result on ADE20K has two components. The first is a gain from the group term over the no-group ablation that is larger than both the random-$W$ gain and the full pairwise gain. The second is a gap between per-teacher matching and mean-target matching that widens as teacher quality becomes more uneven. We expect gains to be larger on ADE20K than on Cityscapes, because more classes means more room for within-group confusion. Finding the opposite would count against the mechanism.
References
- Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels
- Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels
- Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaption
- Transformer-CNN Cohort: Semi-supervised Semantic Segmentation by the Best of Both Students
- Self-Decoupling and Ensemble Distillation for Efficient Segmentation
- Multi-Label Image Recognition With Graph Convolutional Networks
- Semantic-guided Representation Learning for Multi-Label Recognition
- Multi-Label Knowledge Distillation
- Channel-wise Distillation for Semantic Segmentation
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap