When a pretrained language model like BERT is fine-tuned on several tasks at once, the tasks share one internal representation, and some of them can quietly erase knowledge that others, or future tasks, would have used. Structural probing offers a direct view of one kind of such knowledge. A structural probe is a small linear map, fit on top of frozen word vectors, under which distances between words match their distances in a grammatical parse tree. That syntax lives in a small subspace, roughly 64 to 128 directions out of 1024. We propose to use this probe-defined subspace as a training-time anchor. During multi-task training, a penalty keeps word-to-word distances inside the subspace close to their pretrained values, and every other direction stays free to adapt to the tasks.
The question is whether protecting this particular subspace improves transfer compared with protecting nothing, protecting everything, or teaching syntax back through an extra parsing task. The design includes a test of whether the protected geometry is actually used by the network or is only encoded incidentally. It also tests whether the protection holds up on sentences whose content words are nonsense. If anchoring a random subspace of the same size works equally well, the linguistic grounding is irrelevant and we will report that.
Core Claim
Our claim is that multi-task mixtures containing local, token-level tasks, such as part-of-speech (PoS) tagging, erode the syntactic tree geometry of the shared encoder. We claim this erosion predicts poorer transfer to syntax-dependent tasks trained afterwards. We further claim that a label-free distance-preservation penalty restricted to the probe subspace reduces the erosion at a smaller cost to the training tasks than anchoring the whole representation.
Three results motivate the claim. On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations tracked probe scores across single-task fine-tuning of BERT-base. PoS tagging steadily degraded the parse-tree geometry, while semantic role labeling cost about 12 UUAS points early before stabilizing. UUAS, or undirected unlabeled attachment score, is the fraction of tree edges recovered by building a minimum spanning tree from the probe's distances. That study examined only single tasks, so it does not tell us whether training on several objectives jointly prevents the collapse, worsens it, or leaves it unchanged. A Structural Probe for Finding Syntax in Word Representations supplies the method: the linear map $B$, the depth and distance formulation, and the low-rank finding that makes a targeted anchor cheap. Multi-Task Deep Neural Networks for Natural Language Understanding trained BERT jointly on all nine GLUE tasks and treated the shared encoder as a black box. It supplies the training regime we instrument.
Research Program
The first stage is purely diagnostic. We train MT-DNN-style models on BERT-base under several task mixtures. The mixtures are GLUE alone, GLUE plus PoS tagging on English EWT, and GLUE plus dependency parsing. At about 30 checkpoints per run, matching the measurement density of Pérez-Mayos et al., we fit fresh structural probes at layers 6 to 8. At each checkpoint we record UUAS and the Spearman correlation between predicted and gold tree distances (DSpr) on natural EWT.
We also record both scores on Jabberwocky EWT. This is the variant from Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing, in which content words are replaced by pronounceable nonwords. That paper showed probe scores drop by 15.4 UUAS on average when semantics is removed. It constrains us: a model can keep natural-text UUAS while losing grammar-only structure, so every retention claim is reported on both versions.
The second stage is the intervention. Before multi-task training begins, we fit a rank-64 distance probe $B$ on the pretrained encoder at the chosen layer. After that, no treebank labels are used. The anchor penalty is computed on the tasks' own unlabeled input sentences:
$$\mathcal{L}_{\text{anchor}} = \frac{1}{|P|}\sum_{(i,j)\in P}\Big(\|B(h_i-h_j)\|^2 - \|B(h^0_i-h^0_j)\|^2\Big)^2$$
Here $h_i$ is the current representation of word $i$, $h^0_i$ is the frozen pretrained model's representation of the same word, and $P$ is the set of word pairs in a sentence. The total loss is the usual weighted sum of task losses plus $\lambda\,\mathcal{L}_{\text{anchor}}$.
Because the penalty only constrains 64 projected directions, the encoder can still move freely in the other directions. This is the specific bet that separates the approach from anchoring the whole representation. It also differs from the gradient-subspace rotation in RotoGrad: Gradient Homogenization in Multi-Task Learning, which rotates shared features per task to align gradients. RotoGrad's subspaces come from the optimization and carry no linguistic meaning, whereas ours is fixed by a probe with a known interpretation.
The third stage tests whether the preserved geometry matters to the network. Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? showed that probes reach about 80% accuracy on properties that were useless to the training task. High probe scores after anchoring would therefore prove nothing on their own. Our utility measures are downstream instead. First, we fine-tune a dependency parser from each multi-task checkpoint on reduced-size UD treebanks. Second, we measure CoLA, the GLUE grammaticality task. Third, we repeat MT-DNN's low-data domain adaptation to SNLI and SciTail. We then ask whether the early-training drop in Jabberwocky UUAS predicts, across runs and mixtures, the later loss on these transfer measures.
Baselines
The naive reference point is vanilla joint training with MT-DNN's recipe and no anchor. MT-DNN reports an overall GLUE score of 82.7%, compared with 80.5% for single-task BERT-large (Multi-Task Deep Neural Networks for Natural Language Understanding). That figure is for BERT-large. We run BERT-base for compute reasons and will reproduce the recipe ourselves, so our comparison number is our own reproduction rather than the reported one. A single-task fine-tuning reference is included so that negative transfer can be counted per task. Beating vanilla training means equal or better GLUE together with measurably better parser transfer and Jabberwocky retention.
The strongest syntax-specific competitor is adding dependency parsing as an auxiliary task. Pérez-Mayos et al. show that parsing sharpens the tree geometry, so this is the obvious way to keep syntax alive. It needs treebank labels throughout training, and it introduces the kind of auxiliary distribution shift that ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning identifies as the real cause of negative transfer. No reported result was found on Lacuna for GLUE multi-task training with an auxiliary parsing task.
The strongest general-purpose competitor is whole-representation anchoring. This means distilling all layer-$\ell$ features toward the pretrained model with the same $\lambda$ search. No reported result was found on Lacuna for this on GLUE.
We also include loss-balanced task weighting (Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning). It rescales each task loss by its current-to-initial loss ratio. On PubChem it reduced the number of tasks with negative transfer from 48 to 42, but it has no reported GLUE number on Lacuna.
The key ablation is a random rank-64 anchor: the same penalty applied through a random orthonormal projection. A second ablation anchors the directions orthogonal to $B$, the complement of the syntax subspace. These isolate whether the syntax directions matter or whether any low-rank constraint would do.
All methods share the same data, the same number of optimization steps, and the same 16-trial hyperparameter budget for their strength parameter, selected on GLUE development sets. Everything is run over five seeds. The proposal wins if it sits on a better frontier than all baselines, defined by two coordinates: average GLUE change from vanilla, and parser-transfer LAS (labeled attachment score) together with Jabberwocky UUAS. The syntax anchor must also beat the random anchor beyond seed variance.
Why not just…?
Why not just rely on vanilla multi-task training, since the GLUE tasks already regularize each other? Joint training might on its own prevent the PoS-induced collapse. Our first diagnostic stage measures exactly this. If the GLUE-plus-PoS mixture keeps natural and Jabberwocky UUAS near pretrained levels, the intervention is unnecessary and the diagnostic itself becomes the contribution. The evidence suggests shared training does not always protect: the ForkMerge paper reports 10–15% drops on DomainNet under standard multi-task learning, and LBTW reports 48 of 128 PubChem tasks degrading.
Why not just add dependency parsing as an auxiliary task? This is the real threat. It may retain syntax better than our anchor. We run it directly and compare on the same frontier. Our anchor has two potential advantages. It needs a treebank only once, to fit $B$, which matters for languages and domains with little labeled data. It also preserves the pretrained geometry rather than pushing the encoder toward one treebank's annotation style, which should matter for transfer to other treebanks. If auxiliary parsing dominates on both axes, we will report the anchor as a cheaper but weaker approximation.
Why not just anchor the whole representation to the pretrained model? Full anchoring should also retain syntax, but it restrains every direction the tasks need to change. The low-rank result of Hewitt and Manning suggests most of the syntax can be protected by constraining a small fraction of the space. The frontier comparison tests this directly. If full anchoring costs no more GLUE accuracy than the subspace anchor, the targeting buys nothing.
Why not just use a loss- or gradient-based remedy such as LBTW or ForkMerge? These methods balance task progress but cannot see which structure in the representation is being lost. ForkMerge also depends on a representative validation set, and no GLUE validation set measures parser transfer. We include LBTW as a baseline. We also test whether the anchor is complementary by combining the two.
Decisive Checks
Three outcomes would settle the proposal. The first is whether the GLUE-plus-PoS mixture reduces Jabberwocky UUAS relative to GLUE alone. The second is whether the early-training drop in Jabberwocky UUAS predicts later parser-transfer loss across runs, with a rank correlation reported with its confidence interval. The third is whether the syntax-subspace anchor beats the random-subspace anchor and full anchoring on the task-accuracy-versus-transfer frontier.
A plausible negative outcome is that GLUE scores are unaffected by syntax loss while parser transfer suffers. That result would itself be informative. It would mean GLUE-based model selection overlooks damage that matters for syntax-dependent downstream work.
References
- On the evolution of syntactic information encoded by BERT's contextualized representations
- A Structural Probe for Finding Syntax in Word Representations
- Multi-Task Deep Neural Networks for Natural Language Understanding
- Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
- ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning
- Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning