Text-Anchored Label Transfer for Multi-Task Learning When Each Dataset Labels a Different Task

Dense scene models are increasingly trained to predict several things at once, such as semantic segmentation and depth. Their training data is usually a patchwork. One dataset has segmentation masks, another has depth maps, and the two were recorded with different cameras, cities, and weather. Current methods for learning from partial labels fill the gaps by moving supervision across tasks: a pattern learned from images that have segmentation labels is used to generate pseudo-labels for images that only have depth labels. Those methods have been tested only on splits of a single dataset, where every image comes from the same visual domain. When each task's labels come from a different environment, "which task is supervised" and "which environment the image comes from" become the same variable. The network can then learn environment cues as a shortcut, and cross-task pseudo-labels carry fog, lens geometry, or color statistics from one dataset into the other.

We propose to route all cross-task label transfer through a feature subspace anchored to fixed language embeddings of the scene categories ("road", "car", "person") taken from a vision-language model. A second, domain-private subspace absorbs everything that identifies which dataset an image came from. A penalty keeps the two subspaces orthogonal. The private features may still help each task inside its own domain, but they cannot be used to manufacture labels for the other domain. The claim we test is that this split removes the harm caused by confounding task labels with environments, and that it improves zero-shot performance in environments seen at no point during training. We test it against strong partial-label methods and against simpler fixes such as style randomization.

Core Claim

The disjoint labeling structure that makes this setting hard also supplies a free resource: every training image carries a domain label, namely the dataset it came from. Domain-separation methods normally need unlabeled target images to learn a private encoder (Domain Separation Networks, DA-Ada). Here the source datasets already differ from one another, so the private subspace can be trained without any target data. The text anchors give the shared subspace a fixed meaning that does not move when the environment changes. Our specific prediction is that the anchored, orthogonalized transfer path matters most when label availability is confounded with domain. It should matter little when labels are spread evenly across domains. Measuring that difference is the central experiment.

Relation to Prior Work

Three partial-label methods supply the problem formulation and the strongest baselines. Joint-Task Regularization projects predictions and targets into a shared latent space at cost linear in the number of tasks. DiffusionMTL refines imperfect multi-task predictions by denoising. Hierarchical Task Tokens (HiTTs) discovers pseudo-labels through learned global and per-category task tokens, keeping the category tokens orthogonal to one another. All three split a single dataset such as NYUD-v2 or Cityscapes. The research file notes that JTR itself lists domain gaps across disjoint datasets as unsolved. We keep the HiTTs token design but change two things. The per-category tokens are fixed text embeddings rather than vectors learned from scratch. The orthogonality is imposed between semantic content and domain identity, which is a different axis from HiTTs' category-versus-category separation.

The work on semantic anchors motivates using language as the invariant reference. LTrack showed that CLIP-derived pseudo-descriptions improve tracking when moving from MOT17 to the unseen MOT20, but it fuses text and visual features by concatenation with no separation. AttMOT concedes that blur degrades semantic attribute extraction. That concession constrains our design: the anchor can only be as invariant as the visual features projected onto it. We therefore measure anchor-subspace domain leakage directly rather than assuming it is zero.

The orthogonality literature supplies the mechanism. Domain Separation Networks and ASP-MTL use a Frobenius-norm difference loss between shared and private codes. ASP-MTL reports that adding it lowers the average error from 14.9% to 13.9% on 16 text tasks. These papers also expose a real conflict. DSN and TFD discard private features at inference, while DA-Ada argues that domain-specific residuals help detect objects in fog. Our design separates the two roles: private features are allowed in each task's own predictions but barred from cross-domain label transfer. We test both the keep and the discard variants. Finally, AH-OCDA supplies a strong single-task approach to unseen environments, Fourier-amplitude curricula plus associative memory. That approach motivates our main "why not just" competitor.

Research Program

We start from a HiTTs-style network: a shared encoder, task decoders for segmentation and depth, and token-based pseudo-label discovery. At each spatial location the encoder feature $f$ is split into two parts. The anchored part is $f_a = P_A f$, where $P_A$ projects onto the span of the category anchors $W t_1, \dots, W t_C$. Here $t_c$ is the frozen CLIP text embedding of category $c$ and $W$ is a learned linear map into feature space. The private part is $f_d = g(f)$, produced by a small separate head $g$. Training uses the supervised loss on whichever task is labeled for an image, plus one penalty and two classifiers:

$$\mathcal{L} = \mathcal{L}_{\text{sup}} + \lambda_{\perp}\,\| F_a^{\top} F_d \|_F^2 + \lambda_{d}\,\mathcal{L}_{\text{dom}}(F_d) - \lambda_{\text{adv}}\,\mathcal{L}_{\text{dom}}(F_a)$$

$F_a$ and $F_d$ stack the anchored and private features over a batch. The middle term is the DSN-style orthogonality penalty. $\mathcal{L}_{\text{dom}}$ is the cross-entropy of a classifier predicting the source dataset: we minimize it on the private features and maximize it on the anchored features through a gradient-reversal layer.

Pseudo-label discovery is the HiTTs inter-task and fine-grained token attention, and it reads only $f_a$. The task decoders read either $f_a$ alone or the concatenation of $f_a$ and $f_d$, which is the keep-versus-discard question. Because the per-category tokens are text embeddings, segmentation classes and depth structure share a vocabulary. For example, the depth decoder learns how far away "road" typically appears from images carrying only depth labels. The segmentation side learns where "road" is from images carrying only segmentation labels. Unseen environments can also be described by category vocabulary without any images from them.

The analysis has three parts. First, we measure how much damage confounding does to existing methods. Second, we test whether anchoring and orthogonality each reduce that damage. Third, we track the domain decodability of $f_a$ with a linear probe throughout training, which shows whether the separation actually holds or only lowers the loss.

Baselines

The naive reference point is loss masking: one shared network trained on the pooled datasets, computing each image's loss only on the task it has labels for. DiffusionMTL reports that this can underperform single-task models. We also train single-task models per task, and we use fully supervised in-domain models as ceilings.

The strongest existing competitor is HiTTs. It is the most recent partial-label method in the file, it is closest in architecture to ours, and it reports a +16.91% multi-task gain over single-task baselines on NYUD-v2. JTR is a second strong competitor, reporting +21.92% relative improvement over the supervised baseline under random labels on NYUD-v2. That number is measured against a different reference, so we will not compare the two directly. We also include DiffusionMTL, which reports +3.65% on PASCAL-Context one-label. In the domain-confounded setting below, no reported result was found on Lacuna for HiTTs, JTR, or DiffusionMTL, and no Cityscapes number for HiTTs or JTR. We will therefore run all three from their released designs.

Fairness rules apply to every method. All share the same backbone, the same pooled images and label masks, and equal training iterations. Each gets the same hyperparameter search budget, and the search is selected on a held-out split of the source domains only, never on the unseen environments.

Beating HiTTs means two things, both measured as means over three seeds. Our model must have a higher multi-task gain on the unseen environments. It must also have a smaller confound gap, defined below, with the improvement exceeding seed-to-seed spread.

The ablations isolate each ingredient. One replaces the text anchors with HiTTs' learned orthogonal tokens. One removes the anchor-versus-private orthogonality term and the adversarial term. One lets pseudo-label discovery read the private features. One drops the private branch at inference. Finally, for DA-Ada, which needs target images, we report it as an oracle that sees unlabeled target images, not as a fair competitor.

Why not just…?

Why not just run HiTTs or JTR on the pooled data? That is the first thing we do, and it is a real possibility that they cope well. The paired protocol described below measures this directly. If the confound gap for HiTTs is near zero, the premise of the proposal fails, and we would report that, possibly to the field's benefit.

Why not just randomize style so that domain no longer predicts which task is labeled? Fourier-amplitude swapping, the operation behind AH-OCDA's curriculum, could break the confound cheaply for weather-like shifts. This is the most serious threat. We test it by adding amplitude mixing to HiTTs as a baseline and to our method as a combination. We also include a domain pair where the shift is mainly camera geometry and field of view (Cityscapes versus KITTI). Amplitude swapping cannot simulate that kind of shift. If style randomization matches our method on both shift types, we will conclude that orthogonal anchoring is unnecessary. If it matches only on the fog pair, the result marks where the method is needed.

Why not just use frozen CLIP features, which are already fairly domain-invariant? The ablation with text anchors but no orthogonality answers this. ASP-MTL and DSN both found that removing the difference loss hurts transfer, but neither tested this regime. Our linear probe also shows whether frozen CLIP features still encode the source dataset; if they do not, the orthogonality term should add little, and we will report that.

Why not just learn the category tokens, as HiTTs does? Learned tokens are free to absorb domain identity. Fixed text anchors are not. The learned-token ablation measures how much the language grounding adds beyond the orthogonality term.

Proposed Experiments

The main protocol pairs two training setups with identical label counts. In the domain-confounded setup, Cityscapes images from one group of cities carry only segmentation labels, and Foggy Cityscapes images from a disjoint group of cities carry only disparity labels. In the domain-balanced control, the same numbers of segmentation and disparity labels are assigned at random across both pools. The confound gap is a method's multi-task gain in the balanced setup minus its gain in the confounded setup. A second pair repeats this with Cityscapes segmentation and KITTI depth, which is a geometry shift. Evaluation on unseen environments uses C-Driving (BDD100K) for segmentation mIoU. The AH-OCDA result of 31.4 mIoU from GTA5 to C-Driving is reported only as context, because it is a different single-task, adaptation-based setting. For depth, KITTI is held out from the first pair and evaluated with absolute relative error and RMSE. In-domain NYUD-v2 one-label and random-label runs check that we do not lose ground in the standard setting.

The decisive checks are as follows. Existing methods should show a clearly positive confound gap; without one, the premise fails. Our method should shrink that gap and raise multi-task gain on the unseen environments relative to HiTTs with amplitude mixing. The linear probe should show the domain decodability of $f_a$ falling toward chance during training. If decodability falls but performance does not improve, the penalty is working but domain leakage was not the cause of the problem. The keep-versus-discard comparison will settle the DSN versus DA-Ada disagreement for this setting: we expect keeping private features to help each task's own domain and discarding them to help the unseen environments. Finally, a sweep over $\lambda_{\perp}$ will report the sensitivity that DSN warns about, so readers can see whether the gains depend on a narrow setting.