When several segmentation datasets are pooled to train one model, the datasets differ in two ways that the literature usually treats as one. They look different: synthetic driving footage, indoor scans and rainy dashcam frames. They are also labeled differently. One dataset's "oven" includes the stovetop, another's does not, and one dataset splits "rider" from "person" while another merges them. Methods that give each dataset its own "expert", meaning a dedicated sub-network, cannot tell which of these differences they are absorbing.
This note proposes a model that keeps the image-understanding part fully shared and gives each dataset only a small learned translator. The translator converts predictions over one unified class list into that dataset's native labels. The central experiment is a controlled factorial study that asks where dataset-specific capacity actually pays off: in the visual features or in the label mapping.
The practical stakes are concrete. The winner of the Robust Vision Challenge 2022 used a single model over a hand-merged 256-class vocabulary that its authors call "naive and noisy". The runner-up found that standard domain-alignment methods made things worse. Neither tried dataset-specific capacity. If annotation convention, rather than appearance, explains most of what per-dataset experts buy, then multi-dataset systems should spend their specialization budget on the label interface. Doing so is cheaper, easier to interpret, and still yields a single universal prediction.
Core Claim
In multi-dataset semantic segmentation with fragmented label vocabularies, a large share of the benefit that per-dataset experts provide comes from absorbing annotation conventions rather than visual domain differences. A shared backbone with a unified-class head, plus a per-dataset, context-dependent label-mapping module, should match or beat equally sized per-dataset feature experts on native-label accuracy. It should also beat them on held-out datasets and on unified-vocabulary predictions.
For dataset $d$, pixel $i$ with shared features $h_i$, and unified-class probabilities $q(u \mid h_i)$, the native-class prediction is
$$ p_d(c \mid h_i) = \sum_{u} A_d(c, u \mid h_i)\, q(u \mid h_i), \qquad A_d(\cdot, u \mid h_i) = \mathrm{softmax}_c\big(M_d(\cdot,u) + g_d(h_i)(\cdot,u)\big). $$
Here $c$ ranges over dataset $d$'s native classes plus an "unlabeled" class. $M_d$ is a mapping matrix initialized from the hand-made many-to-one mapping. $g_d$ is a small low-rank network that lets the mapping depend on context; for example, a unified "oven" pixel can be read as "stove" when it sits on a countertop. Training uses only native-label cross-entropy through $p_d$. The unified head $q$ is shared by all datasets and is what the model outputs when no dataset convention is requested.
Relation to Prior Work
Four groups of prior work shape this proposal.
The first group motivates it. Two Robust Vision Challenge entries merge six to nine datasets into a unified vocabulary, 256 classes in 1st Place Solution of The Robust Vision Challenge (RVC) 2022 Semantic Segmentation Track and 194 in An Empirical Study on Multi-domain Robust Semantic Segmentation. Both train one monolithic model and apply a fixed many-to-one reverse mapping at test time. The second of these also reports that DANN, MMD and IRM alignment degraded results. We keep their unified-vocabulary setting but replace the fixed reverse map with a learned, context-dependent one, and we add the missing comparison against dataset-specific capacity.
The second group supplies the multi-expert baselines against which we test. Decoupled Training pairs a shared backbone with per-domain heads trained in three phases. Best Sources Forward and CoDEx route test samples to source-specific experts. COME mixes shared and source-specific experts inside the network. All of them assume harmonized labels, so any specialization they learn is attributed to "domain". Our design explicitly splits specialization into a label-side part and a feature-side part.
The third group is the closest prior work, Simple multi-dataset detection, known as UniDet. It trains per-dataset detection heads on a shared backbone and then merges classes offline with a 0-1 integer program over how often heads fire together on the same images. UniDet discovers the taxonomy after training and treats class correspondence as all-or-nothing. We learn soft, context-dependent correspondences during training, and we use them as the only place where dataset-specific parameters live.
The fourth group constrains the design. TACS shows that stochastic label mapping helps across coarse-to-fine and partially overlapping taxonomies, which suggests our mapping should be soft. Taxonomy-Structured Domain Adaptation warns that imposed hierarchies mislead when they do not match the visual data, which argues against hard-coding WordNet structure. A Sober Look at the Unsupervised Learning of Disentangled Representations and BARS show that seeds and tuning budgets can erase claimed gains. That finding dictates the evaluation protocol below.
Research Program
We build one training and evaluation pipeline in which the only thing that varies is where dataset-specific parameters sit. The shared part is a SegFormer-style backbone and a unified-vocabulary head over the 194-class taxonomy of the runner-up entry.
The design has four cells. The first has no dataset-specific parameters and uses a fixed reverse map, which is the monolithic recipe. The second is label-side only, the proposed model. The third is feature-side only: per-dataset low-rank adapters in the backbone, or a COME-style mixture of shared and per-dataset experts, still read out through the fixed reverse map. The fourth combines label-side and feature-side capacity. The label-side and feature-side cells receive the same number of dataset-specific parameters.
All cells use the same data, the same hardware budget, the same schedule, the same class-balanced sampling and warmup, and the same number of tuning trials. The runner-up entry reports that class-balanced sampling with warmup alone was worth nearly 1% mIoU, which is why these must be held fixed. Every configuration runs with at least three seeds. Evaluation uses one frozen metric implementation for native-label mIoU on each Robust Vision Challenge dataset: ADE20K, Cityscapes, Mapillary Vistas, ScanNet, VIPER and WildDash 2.
Because the natural datasets confound appearance with convention, we also construct two controlled shifts from the same source data. In the convention-only shift, Cityscapes is split into two halves. One half is relabeled by rule: classes merged, a class split by context, and boundary regions reassigned to mimic the "oven includes stovetop" kind of disagreement. The images are statistically identical, so any gain from specialization must come from convention. In the appearance-only shift, VIPER, Cityscapes and WildDash are restricted to their fully agreeing shared classes, so the labels agree and only appearance differs. The prediction is that label-side capacity wins clearly on the first shift, feature-side capacity wins on the second, and the natural benchmark sits closer to the first.
The learned mappings $M_d$ and $g_d$ are also inspected. For each dataset pair, we list the unified classes whose learned mapping departs most from the hand-made one. We then check these against a small manual audit and against UniDet-style co-firing statistics computed on the same model. This turns the translator into a tool for auditing where the taxonomies actually disagree.
Baselines
The naive reference is per-dataset training: one model per dataset under the same schedule. It gives the ceiling for native-label accuracy at no sharing and shows whether pooling helps at all.
The strongest existing method is the monolithic unified-vocabulary model with a fixed many-to-one reverse map, as in 1st Place Solution of The Robust Vision Challenge (RVC) 2022. That paper reports 43.46 mIoU on ADE20K and 82.0 mIoU on Cityscapes, using nine training datasets and 64 GPUs. We cannot match that compute. We therefore reimplement the recipe inside our budget, which matches the runner-up's 4-V100 regime; for An Empirical Study on Multi-domain Robust Semantic Segmentation no reported per-dataset test mIoU was found on Lacuna. The published numbers serve as an external sanity reference, not a direct target. Beating this baseline means a mean native-label mIoU gain across the six datasets that exceeds the seed-to-seed spread, with no dataset losing by more than that spread.
The strongest multi-expert competitor is a per-dataset-head model trained with the three-phase schedule of Decoupled Training, adapted to segmentation. That paper reports 77.3% average accuracy on DomainNet and 61.0% on FMoW in classification. We know of no reported segmentation number for it. We give it the same parameter budget as our translator and the same tuning trials. Beating it means matching it on in-distribution native mIoU and exceeding it on the held-out and low-data tests described below.
A feature-side expert baseline, per-dataset adapters or a COME-style mixture as in COME, fills the third cell. For reference, COME reports 0.6587 mean AP in its inter-organ ultrasound setting. The key ablation drops $g_d$, leaving only a learned static matrix $M_d$. This isolates whether context-dependent mapping matters, or whether simply learning the reverse map is enough.
Why not just…?
Why not just scale the monolithic unified model, as the challenge winner did? Scale is a real threat, since the winner beat prior entries by roughly ten points on ADE20K. However, that model still bakes the hand-made mapping errors its authors acknowledge into both training targets and test-time outputs, and no amount of scale can learn around a wrong fixed reverse map. We test this directly: the static-matrix ablation versus the fixed map at equal scale shows whether mapping errors matter. The convention-only shift shows whether a larger shared model can compensate. If the gap closes as backbone size grows across two sizes, we will report that the translator is a small-model fix.
Why not just give each dataset its own output head? This is the most serious alternative, and it may match the proposal on in-distribution native mIoU, because a free head can absorb any convention. The differences we expect lie elsewhere. First, free heads produce no shared unified prediction, and they learn overlapping classes separately, which should hurt small datasets such as WildDash 2. Second, they cannot label a held-out dataset. We train on five datasets, hold out the sixth, and give our model only the hand-made mapping for it. The per-head model must borrow the closest dataset's head. If per-dataset heads match us on both the held-out and small-dataset tests, the method claim fails, and the paper reduces to the diagnostic result about convention versus appearance.
Why not just add dataset-specific experts inside the backbone, as mixture-of-experts methods do? Such experts spend capacity on appearance, and when labels disagree they also silently absorb convention, at a higher parameter cost and with routing that degrades on unfamiliar inputs, as the CoDEx authors note. The parameter-matched factorial comparison, together with the appearance-only shift, measures exactly how much such experts add once convention is handled at the output.
Why not just fix the unified taxonomy by hand, or merge it with UniDet's integer program? A better static taxonomy is plausible and cheap. We therefore include a UniDet-style co-firing merge, solved on our own model, as an additional fixed-map baseline. Neither a hand-made nor a co-firing merge can express context-dependent correspondence of the stovetop kind, and the $g_d$ ablation measures how much that matters.
Decisive Checks
Three results would settle the question.
The first is the convention-only Cityscapes split. Label-side capacity should recover nearly the per-convention upper bound, while feature-side experts of equal size should not. If feature-side experts do just as well, the core claim is wrong.
The second is the natural six-dataset benchmark. If the label-side cell captures most of the gain of the combined cell, meaning more than half of the improvement from the monolithic baseline to the both-sides cell, this supports the claim that the specialization budget belongs at the label interface.
The third is the held-out-dataset and WildDash 2 low-data tests against per-dataset heads. These decide whether the translator is a genuinely better design or merely an equally good and more interpretable one.
All comparisons are reported with seed spreads under the frozen pipeline, so that a result smaller than seed noise is reported as a null result.
References
- 1st Place Solution of The Robust Vision Challenge (RVC) 2022 Semantic Segmentation Track
- An Empirical Study on Multi-domain Robust Semantic Segmentation
- Decoupled Training: Return of Frustratingly Easy Multi-Domain Learning
- Best sources forward: domain generalization through source-specific nets
- CoDEx: Combining Domain Expertise for Spatial Generalization in Satellite Image Analysis
- COME: Dual Structure-Semantic Learning with Collaborative MoE for Universal Lesion Detection Across Heterogeneous Ultrasound Datasets
- Simple Multi-dataset Detection
- TACS: Taxonomy Adaptive Cross-Domain Semantic Segmentation
- Taxonomy-Structured Domain Adaptation
- A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
- BARS: Towards Open Benchmarking for Recommender Systems