When one model is trained on several related tasks at once (multi-task learning), a consistent pattern appears whenever those tasks differ in how finely they carve up their labels. Fine-grained supervision, such as eleven kinds of sexism, word senses, or continuous similarity scores, helps the coarse task. Coarse supervision, such as sexist or not, broad event types, or binary similar/dissimilar, does not help the fine task and often hurts it. This "one-way street" has been reported in argument mining, semantic role labeling, event detection and text similarity. It is usually read as a statement about information: coarse labels simply carry too little detail. This proposal argues that the reading is partly wrong. A coarse label on an example that has no fine label still carries real information about the fine label, because it rules out every fine class outside its category. The failure lies in how coarse supervision is usually attached to the network. The standard design gives the coarse task its own output layer on a shared encoder. That layer can be satisfied by squashing the distinctions inside each coarse category, and nothing in the coarse loss objects.
The proposal removes the separate coarse output layer entirely. The coarse prediction is computed as a sum of the fine head's probabilities: over the fine classes the coarse label allows when the hierarchy is known, or through a small learned mixing matrix when it is not. Coarse gradients then reach the shared encoder only through the fine output space. In that space the gradient raises the fine classes the coarse label permits, in proportion to the model's own current fine-grained beliefs. The prediction is concrete. In the regime where coarse labels cover many more examples than fine labels, which is the usual one in practice (EDOS labels all posts as sexist or not but assigns fine vectors only to the sexist ones), this "coarse-through-fine" design should make coarse data improve the fine task. Both separate-head joint training and asymmetric gradient surgery should fail to do so.
Core Claim
Let $p_\theta(f\mid x)$ be the fine head's softmax over fine classes $f$ for input $x$, with parameters $\theta$. Let $S(c)$ be the set of fine classes compatible with coarse label $c$. For an example that carries only a coarse label, the coarse loss is the negative log of the probability mass the fine head places on the admissible set:
$$\mathcal{L}_{\text{coarse}}(x,c) = -\log \sum_{f \in S(c)} p_\theta(f\mid x).$$
The gradient of this loss with respect to fine logit $z_f$ is $p_\theta(f\mid x) - q(f)$. Here $q$ is the fine posterior renormalized inside $S(c)$ and set to zero outside it. The coarse loss therefore suppresses inadmissible fine classes and redistributes their mass across the admissible ones according to the model's current fine-grained ranking. This is a form of self-training confined to the admissible set, and it gives the network no incentive to merge fine classes within a category. A separate coarse head behaves differently. Its loss is indifferent to within-category structure, so the shared encoder is free to compress that structure away. When the hierarchy is not nested, as between word senses and event types, the sum is replaced by $p(c\mid x)=\sum_f A_{cf}\,p_\theta(f\mid x)$ with a learned non-negative matrix $A$ whose columns sum to one.
The claim has three parts, and each can be tested separately. First, coarse-only data improves the fine task under this design and does not under separate heads. Second, the gain disappears when no extra coarse-only data exists, which pins the benefit to label coverage rather than to an optimization trick. Third, the degradation seen with separate heads comes mainly from within-category feature collapse, not from conflicting gradient directions. The third part speaks directly to the unresolved dispute over whether negative transfer is a gradient-conflict problem or a representation-shift problem.
Relation to Prior Work
Several papers motivate the proposal by documenting the one-way street. In the argument-mining study, low-level argument units improve rhetorical tasks, with citation context F1 rising from 46.1 to 49.9, and nothing flows back (Lauscher et al.). Relation classification helps semantic role labeling but not the reverse (Guo et al.). Fine similarity scores transfer down to binary decisions but binary labels do not transfer up (Boyd-Graber et al.). All three attach each granularity through its own head and read the asymmetry as a fact about the tasks. We test whether it is instead a fact about that attachment.
The sharpest motivating result comes from Chride at SemEval-2023 Task 10. That system combines binary, 4-category and 11-vector cross-entropy losses on a shared DeBERTa-V3. The binary loss combined with the 11-way loss breaks training, while the 4-category loss combines well with it, and the paper gives no mechanism. Our account predicts exactly this pattern. The binary head tolerates the largest within-category collapse; the 4-category head, being closer in granularity, tolerates less. Routing both through the 11-way head should remove the incompatibility.
Other work constrains the design. ForkMerge shows that negative gradient cosine does not by itself predict negative transfer, and attributes harm to auxiliary data pulling representations off the target distribution. This warns us not to build another gradient-projection method. Our mechanism works at the level of representations and label structure, and our diagnostics measure both gradient cosine and within-category collapse so that the two explanations can be told apart. Hierarchical Granularity Transfer Learning shows that basic-level training erases sub-category detail, and reports a large gap between basic-level accuracy (about 93%) and sub-level accuracy. This supports the collapse hypothesis. BigSPN closes the gap with attribute descriptions and a second encoder; we use neither.
The closest methods for asymmetric transfer handle asymmetry differently. Deep-AMTFL assigns direction by training loss, so reliable tasks reconstruct shared features. SAAL clones source tasks through target decoders, at quadratic cost and with a 6–8× slowdown in its meta-learned variant. Neural representation matching keeps two full networks and pulls event-detection features toward word-sense features with a squared-error penalty. All three treat the coarse task as something to protect or to isolate. None converts coarse labels into supervision for the fine output space, and none adds parameters in proportion to the number of tasks the way SAAL does. Our change is one aggregation step with at most a $C\times F$ matrix. Symmetric gradient surgery, meaning PCGrad and RotoGrad, supplies baselines, not method.
Research Program
We build the coarse-through-fine head in three variants. The first uses a known nested hierarchy with fixed $S(c)$. The second learns $A$ with an entropy penalty that keeps each fine class tied to few coarse classes. The third is a hybrid that keeps a separate coarse head for inference but trains the encoder only through the aggregated fine path. The hybrid lets us check whether the benefit is in training or in inference.
Alongside accuracy we track two diagnostics during training. The first is within-category separability: how well a linear probe on frozen encoder features recovers fine labels inside each coarse category. The second is the cosine between coarse and fine gradients on shared parameters. If separate-head degradation coincides with falling probe accuracy while gradient cosine stays flat, that favors ForkMerge's representation-shift view over the gradient-conflict view. The reverse result would support the conflict view, which would argue against our mechanism.
The main experimental regime varies the fraction of examples that carry fine labels while every example keeps its coarse label. This is where the coarse task can contribute anything. We also run a control in which coarse labels appear only on the fine-labeled examples, so the coarse labels are redundant.
Baselines
The naive reference points are two. The first is single-task training of the fine head on fine-labeled data only. The second is the same fine head with coarse predictions obtained by summing its probabilities at test time, with coarse data never used in training. The second reference isolates whether coarse-only data contributes at all.
The standard competitor is hard parameter sharing with separate coarse and fine heads and summed losses, as in the Chride hierarchical loss. It is the design practitioners actually use, and the Chride paper documents that it breaks down. On EDOS, the task overview SemEval-2023 Task 10 reports top macro-F1 of 0.8746 on Task A, 0.7326 on Task B and 0.5606 on Task C. Beating 0.5606 on Task C with a single DeBERTa-V3 model is the headline external target.
The strongest existing method for protecting a target task from auxiliary data is ForkMerge. It is run with the fine task as the target and the coarse task as auxiliary, using the same validation split for its mixing search that we use for early stopping. Its reported relative gain over single-task training is 4.03% on NYUv2, and no reported result was found on Lacuna for ForkMerge on EDOS, CIFAR-100 superclasses or ACE 2005.
We also run PCGrad in two forms. One is symmetric. The other is one-sided: only the coarse gradient is projected off the fine gradient. The one-sided form is the obvious "asymmetric gradient surgery" idea. For the non-nested setting we compare against representation matching, which reports 71.2 F1 on ACE 2005 event detection with a CNN against 67.6 for separately trained models (Lu and Nguyen).
Every method uses the same backbone, data, number of optimizer steps and a matched hyperparameter search, for example 20 trials per method on the same validation split. The ablation that isolates our key ingredient is the separate-head model with its encoder gradient cut off, which trains the coarse head only on frozen features. We win only if coarse-through-fine beats every baseline on fine-task macro-F1 by more than seed variance across five seeds in the low-fine-label regime, without losing coarse-task accuracy.
Why not just…?
Why not just stop coarse gradients from reaching the encoder? This is the cheapest way to enforce "fine helps coarse, coarse never hurts fine," and it will likely match us on protecting the fine task. It also throws away whatever coarse-only data could teach the fine task. So it cannot exceed the fine-only single-task reference, while our claim predicts that we do. The stop-gradient ablation settles this directly. If it matches coarse-through-fine on fine-task F1, then the only useful thing about coarse labels is keeping them away from the encoder, and our central claim is false.
Why not just down-weight the coarse loss? Down-weighting addresses magnitude dominance, the mechanism emphasized by Wang et al.. It does not change the direction in which a separate coarse head pulls the features. The sweep over the separate-head coarse weight is part of the tuning budget. If some weight reaches our fine-task F1, magnitude was the whole problem and we will report that.
Why not just apply one-sided gradient surgery? Projecting only the coarse gradient is the natural granularity-aware PCGrad. ForkMerge's finding that gradient conflict does not track harm predicts it will help little, since it removes conflict without preventing collapse. The within-category probe decides the matter. If one-sided PCGrad preserves fine separability and matches us, the gradient view wins in this setting.
Why not just use ForkMerge? ForkMerge can only decide how much of the coarse-trained parameters to keep. When coarse training damages fine structure, its best choice is to keep little of it, which recovers single-task performance but gains nothing. It is a real threat if coarse-only data helps even under separate heads. The low-fine-label sweep will show whether that is so.
Proposed Experiments
The primary benchmark is EDOS. Its three nested levels (binary, 4 categories, 11 vectors) come with a known hierarchy, and the fine labels cover only part of the data, so the coarse-only regime exists naturally. We additionally subsample Task C labels to 25% and 50% of their natural coverage.
The second benchmark is CIFAR-100 with its 20 superclasses as the coarse task. Fine labels are kept on 10%, 25% and 100% of the training set, and superclass labels on all of it. Here the regime is controlled and the images are cheap to train on. The CIFAR-100 routing-network numbers reported for PCGrad (77.5% against 67.7%) come from a different task split, and we do not compare against them.
The third benchmark is the non-nested stress test, ACE 2005 event detection paired with Semcor word senses, with a learned $A$ mapping senses to event types. It asks whether routing event detection through a sense-level layer on one backbone can approach or beat the 71.2 F1 that representation matching gets from two backbones.
The first decisive check is whether EDOS Task C macro-F1 and CIFAR-100 fine accuracy rise with coarse-only data under coarse-through-fine, and fall or stay flat under separate heads. The second is whether that gain vanishes in the redundant-coverage control. The third is whether the probe shows separate heads losing within-category separability exactly where they lose fine accuracy, while gradient cosine does not separate helpful from harmful runs.
If the first check fails, the one-way street reflects real information limits, and we will report it as a negative result. If the first holds but the third favors gradient conflict, then the mechanism needs revising even though the method works.
References
- Investigating the Role of Argumentation in the Rhetorical Analysis of Scientific Publications with Neural Multi-Task Learning Models
- A Unified Architecture for Semantic Role Labeling and Relation Classification
- Bayesian Supervised Domain Adaptation for Short Text Similarity
- Chride at SemEval-2023 Task 10: Fine-tuned Deberta-V3 on Detection of Online Sexism with Hierarchical Loss
- ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning
- Hierarchical Granularity Transfer Learning
- Deep Asymmetric Multi-task Feature Learning
- Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries
- Similar but not the Same - Word Sense Disambiguation Improves Event Detection via Neural Representation Matching
- Gradient Surgery for Multi-Task Learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- SemEval-2023 Task 10: Explainable Detection of Online Sexism
- Granularity-Aware Adaptive Classifier Expansion via Zero-Shot Learning