Parsers for most of the world's languages must be built from a handful of annotated sentences, a treebank in a related language, and raw text in the target language. One training technique uses that raw text well: virtual adversarial training, or VAT. VAT pushes each input slightly in the direction that would most change the model's prediction, then penalizes the model for changing its mind. It needs no labels. Spa showed it helps dependency parsing (predicting which word each word attaches to) when labels are scarce: 92.82 against 92.11 unlabeled attachment score (UAS) with 10% of the English Penn Treebank. The obstacle is cost. Finding the worst-case push requires extra forward and backward passes through the whole network. With a large multilingual encoder such as XLM-RoBERTa, the kind that current cross-lingual parsers depend on, those passes dominate training time.
We propose to apply the push at an intermediate layer of a mostly frozen multilingual encoder rather than at its input. The frozen bottom layers are computed once and cached for all unlabeled target sentences. The worst-case search and the smoothness penalty then only pass through a few adapter-equipped top layers and the tree-structured parser head. We measure the penalty exactly over all parse trees with Spa's dynamic program. The claim is that this recovers most of the gain of full-depth VAT in cross-lingual, low-resource parsing, at a small fraction of its compute. The claim also predicts where the gain should be largest: in target languages whose word order differs most from the source, because their representations land furthest from where the parser was trained.
Core Claim
Current work either makes VAT exact for trees or makes robustness cheap for flat classifiers, and no work does both. Spa solved the output-side cost of VAT for parsing. It computes the KL divergence between two tree-structured distributions in $O(n^3)$ with a cross-entropy semiring, and it sharpens the distributions with temperatures. Spa still perturbs word inputs and backpropagates through the whole encoder. It was tested only on monolingual English with a BiLSTM. Separately, Robustness Tokens and robust prefix-tuning keep a large backbone frozen and train only a few added parameters. These methods have been shown only on image classification, image segmentation and sentence classification. Neither was built for tree outputs or scarce labels. Robustness Tokens recovers DINOv2-G accuracy under attack from 0.3% to 63.1% within 200 optimization steps.
The cross-lingual parsers closest to our setting use "adversarial" in a different sense. Representation Alignment and Adversarial Networks and APGN train a language or domain discriminator through a gradient reversal layer. Both fine-tune the full encoder, and neither makes the model smooth around individual target inputs. Our proposal differs from all of these in one specific way. It asks at what depth the structured smoothness penalty must be applied to help cross-lingual transfer. It then puts the penalty at the shallowest point in the network that still captures the benefit, so the expensive inner loop never touches the frozen layers.
Concretely, let $h(x)$ be the cached output of frozen layer $\ell$ for an unlabeled target sentence $x$. Let $g_\phi$ be the trainable top $k$ adapter layers plus the tree-structured parser head (a biaffine TreeCRF), with parameters $\phi$. Let $p_\phi(\cdot \mid h)$ be its distribution over trees, sharpened by Spa's temperatures. The training objective is
$$\mathcal{L}(\phi) = \mathcal{L}_{\text{sup}}(\phi) + \lambda\, \mathbb{E}_{x}\, \mathrm{KL}\big(p_\phi(\cdot \mid h(x)) \,\|\, p_\phi(\cdot \mid h(x) + r_{\text{adv}})\big).$$
Here $\mathcal{L}_{\text{sup}}$ is the parsing loss on source-treebank sentences and the few annotated target sentences. $\lambda$ weights the penalty. The perturbation is $r_{\text{adv}} = \epsilon\, \nabla_r \mathrm{KL}/\|\nabla_r \mathrm{KL}\|$, estimated with one gradient step from a small random start. The KL itself is computed exactly by the semiring. Every pass in the inner loop costs roughly $k/L$ of a full-encoder pass, where $L$ is the encoder depth.
Research Program
We build a single parser: a frozen XLM-R-sized encoder, lightweight adapters in its top $k$ layers, and a biaffine TreeCRF head. The perturbation depth $\ell = L - k$ is the variable under study. At $k = L$ with full fine-tuning, the method becomes Spa-style VAT applied to a modern encoder. At $k = 0$, only the head is smoothed.
The program has three parts. The first is a depth-versus-cost curve. For each $k$ we record target-language accuracy and wall-clock GPU time, and we check whether accuracy flattens well before full depth. The second is a mechanism analysis. We ask whether the gain from structured smoothness over target raw text grows with word-order distance from the source. This is testable because Meng et al. found gains correlated with word-order gap at $r = 0.938$, and IWR-KD found accuracy anti-correlated with it at $r = -0.88$. The third isolates what does the work: the adversarial direction versus random noise, and the exact structured KL versus a per-word head-distribution KL.
Robust prefix-tuning found that tuning bottom layers mattered more than tuning top layers for defending against attacks. That finding cuts against our preference for shallow perturbation. We treat it as a hypothesis to test rather than dismiss. Our goal is generalization to a new language, not resistance to an attacker, and these two goals may favor different depths. The depth sweep settles which way it goes for parsing.
Baselines
The naive reference point is supervised training alone, using the same frozen encoder, adapters and head. It trains on the source treebank plus the few target sentences and uses no unlabeled text. Every other method has to beat this.
The strongest existing published system on a directly comparable setting is Representation Alignment and Adversarial Networks. It is a fully fine-tuned XLM-RoBERTa biaffine parser with a dictionary-based alignment network and a language discriminator. It reports 68.98 labeled attachment score (LAS) and 76.81 UAS on Vietnamese VTB from Chinese GSDSimp, beating UDify at 66.00 LAS. We will rerun it on the same UD v2.12 split and use its reported figure as the target. Our full method must match 68.98 LAS within noise, using no bilingual dictionary and markedly less GPU time, to count as a success. Clearly exceeding it would be a stronger result.
The most important method baseline is full-depth VAT: Spa's exact semiring objective with gradients through the entire fine-tuned encoder. It uses the same unlabeled text, the same $\epsilon$ and temperature grid, and the same number of hyperparameter trials. The reference number is Spa's own 92.82 UAS at the 10% PTB split, against 92.11 supervised. We will reproduce that setting as a sanity check before any cross-lingual run. The shallow method "beats" full-depth VAT if it retains at least most of the full-depth gain over the supervised reference at a large reduction in wall-clock time, with both measured on identical hardware.
Two ablations isolate the key ingredient. In the first, random-direction perturbations of equal norm replace the adversarial direction. In the second, a per-word KL on head-attachment scores replaces the exact tree KL.
Self-training on the same raw text is a further baseline, described below. For it, no reported result was found on Lacuna.
Why not just…?
Why not just fine-tune the whole encoder with the few target sentences? This is the default in Representation Alignment and Adversarial Networks. There, plain fine-tuning trails the full model by only 0.89 LAS. It is therefore a real competitor on accuracy, but it ignores the raw target text entirely, and it pays full-encoder cost. The supervised reference and full-depth VAT runs show directly how much unlabeled smoothing adds on top of it.
Why not just self-train on pseudo-labeled target text? This is the most serious threat. It is cheap, uses the same raw text, and we found no reported parsing result for it on Lacuna. We will run it with the same frozen encoder, adapters and compute budget, including confidence filtering. If it matches our method, the honest conclusion is that the benefit comes from the target raw text rather than from adversarial smoothness. We would then report the combination of the two as the contribution instead.
Why not just use tree-morphing augmentation or test-time typological constraints? Vania et al. report +9.3 LAS on North Sámi from subtree morphing alone. Meng et al. report +3.5 UAS on average from constraints derived from the WALS typology database. Both are complementary to our method rather than exclusive of it. Morphing needs target trees, and the constraints need reliable target part-of-speech tags. We will stack each with our method on North Sámi and on Hindi or Urdu-like distant targets to check whether the gains add up.
Proposed Experiments
The first benchmark is the seven low-resource targets from Dynamic Syntactic Feature Filtering: Vietnamese, Wolof, Coptic, Maltese, Tamil, Uyghur and Thai, from English or Chinese sources on UD v2.13. That paper reports average gains of +1.84 UAS and +3.43 LAS. We add Vietnamese VTB on UD v2.12 for the direct comparison with Li et al. We also add the genuinely low-resource North Sámi (141 sentences), Kazakh and Galician from UD v2.2, where Vania et al. report 28.23 LAS for Kazakh and 70.46 LAS for Galician.
For unlabeled target text, we use target training sentences with their trees removed. This keeps the setting closed and reproducible, and we report results for any larger raw corpus separately. We measure UAS and LAS, GPU-hours to convergence, and encoder passes per update.
The decisive checks follow from the claim. First, the depth sweep must show accuracy saturating at small $k$ while cost falls roughly in proportion to $k/L$. If gains appear only at full depth, the idea fails. Second, adversarial perturbation must beat random noise at the chosen depth, and the exact tree KL must beat the per-word KL. Third, across the target languages, the gain over the supervised reference should correlate positively with word-order distance from the source. Finally, our method should beat equal-budget self-training on at least the typologically distant targets.
References
- Spa: On the Sparsity of Virtual Adversarial Training for Dependency Parsing
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- On Robust Prefix-Tuning for Text Classification
- Representation Alignment and Adversarial Networks for Cross-lingual Dependency Parsing
- APGN: Adversarial and Parameter Generation Networks for Multi-Source Cross-Domain Dependency Parsing
- Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing
- Implicit Word Reordering with Knowledge Distillation for Cross-Lingual Dependency Parsing
- A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages
- Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing