Does Scale Make Parsers Recover, or Just Make Fewer Mistakes?

A transition-based parser builds a syntax tree one action at a time, for example "attach this word to that one" or "move to the next word." When it makes one wrong move, every later decision is made from a state it never saw during training. That one error can cascade into many. The standard fix is the dynamic oracle, a training signal that says what the best move is from any state, including broken ones, so the parser learns to recover. A recent result complicates this story. Auto-correction et oracle dynamique : certains effets n'apparaissent qu'à taille réduite found that the benefit of dynamic oracles shrinks as models grow. Their largest model had only about 2.7M parameters. The natural reading is that big models "absorb" error recovery on their own. Few people have checked this with modern pretrained encoders, which are far larger.

This note proposes that the shrinking gain has a simpler explanation that has never been separated out. Larger models make fewer first mistakes, so there is less to recover from. This does not show that they recover any better once a mistake happens. We propose to measure recovery directly by planting identical errors into parsers of different sizes and training regimes and then watching what follows. If recovery ability stays roughly constant while the base error rate falls, dynamic oracles remain worth using exactly where errors are common: low-resource treebanks, shifted domains, and non-projective languages. The flat "scale makes them unnecessary" conclusion would then be wrong.

Core Claim

The end-to-end accuracy gain from dynamic-oracle training can be written as roughly the product of three quantities. The first is how many first errors the parser makes. The second is how much extra damage each error causes beyond what is logically unavoidable. The third is how much of that extra damage dynamic training removes. Comparing final accuracy across model sizes, as prior work does, mixes all three together. Our hypothesis is that scaling the encoder mainly reduces the first factor and leaves the second and third largely intact.

There is a structural reason to expect this. With a pretrained bidirectional encoder, each word's representation is computed before parsing starts and does not depend on the parser's state. A wrong move therefore cannot corrupt the word vectors. It can only change which positions the action classifier looks at: which words sit on top of the stack, and which heads are still available. A bigger encoder makes those vectors better, but it does not teach the small classifier on top what to do from configurations it never saw. Parsers whose state is fed back into a learned composition, as in Greedy Transition-Based Dependency Parsing with Stack LSTMs, have a second channel through which errors propagate. We therefore also predict that the location of state dependence matters more than parameter count.

Measuring Recovery by Injection

Our main instrument is controlled error injection. Take a sentence and a parser $\theta$, and run the static oracle to a gold configuration $c$. Then force a wrong action $a$ whose dynamic-oracle cost $\Delta(c,a)$ is positive. That cost is the number of gold arcs that action $a$ makes impossible to recover. Then let $\theta$ parse greedily to the end. The excess propagation of that injection is

$$E_\theta(c,a) = \mathrm{Loss}_\theta(c,a) - \Delta(c,a),$$

where $\mathrm{Loss}_\theta(c,a)$ is the number of gold arcs the finished tree gets wrong. A perfect recoverer has $E=0$, because it loses only what the injected error made unreachable. This measure does not depend on how often $\theta$ errs naturally, because every model receives the same injected $(c,a)$ pairs, sampled once and shared across conditions. We sample $a$ two ways. One matches the confusion distribution of a reference parser, giving realistic errors. The other is uniform over costly actions, giving stress tests. We stratify by error type (attachment, premature reduce, missing or spurious SWAP) and by how far into the sentence the error occurs.

This builds on the recursive-correction analysis in Tackling Error Propagation through Reinforcement Learning: A Case of Greedy Dependency Parsing. That work used a dynamic oracle to find which of a parser's own errors were propagated, and reported that 51.9% of the errors avoided by its reinforcement-learned parser were propagation cases, against 27% in the supervised corpus. Their method cannot compare parsers fairly when the parsers make different first errors. Matched injection fixes this. We also run their recursive-correction analysis on natural errors as a consistency check.

Research Program

The parser is the arc-hybrid system with SWAP. Its non-swap transitions use the static-dynamic oracle of Arc-Hybrid Non-Projective Dependency Parsing with a Static-Dynamic Oracle, and exploration follows the temperature sampling of Training with Exploration Improves a Greedy Stack LSTM Parser. That paper also supplies the constant-time oracle that makes training cheap. We vary two axes.

The first axis is encoder capacity. It covers randomly initialized BiLSTMs at several widths, in the style of Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations and spanning the range Zhao and Bernard studied, through pretrained transformer encoders from roughly 100M to several hundred million parameters.

The second axis is where parser state enters. One setting uses state-independent token vectors with a small feature-position classifier. The other adds a stack-composition module that recomputes representations from the partial tree.

Each cell is trained with a static oracle and with a dynamic oracle plus exploration, using three seeds.

We then fit the three-factor decomposition per cell. We test whether the dynamic-oracle gain in final accuracy is predicted by first-error rate times measured excess propagation. If it is, the decomposition explains Zhao and Bernard's narrowing gap. If dynamic training stops reducing $E$ at large scale, the "absorption" reading is correct and we will report that. Finally, we vary the base error rate without changing the model by evaluating on harder conditions: small UD treebanks, non-projective languages, and cross-treebank transfer within a language. The hypothesis predicts that the dynamic gain for a large encoder returns there in proportion to the first-error rate.

Baselines

The naive reference point is the same parser trained with a static oracle and greedy decoding. Every other condition is reported as a difference from it, using the same encoder, data, and tuning budget. This baseline answers the practitioner's question of whether the extra training machinery is worth it at all.

The strongest existing method on this question is dynamic-oracle training with exploration as in Training with Exploration Improves a Greedy Stack LSTM Parser. It reports 93.56 UAS on English PTB and 87.65 UAS on CTB-5 with a Stack LSTM. We reimplement it within our system and first confirm that our small-scale Stack LSTM configuration lands near those numbers before any scaled comparison.

For non-projective treebanks, the reference is the static-dynamic oracle of Arc-Hybrid Non-Projective Dependency Parsing with a Static-Dynamic Oracle. It reports gains of 0.5 to 3.5 LAS over static training and 59.53 LAS on Ancient Greek in CoNLL 2017 UD.

For the scale effect itself, we replicate the PETIT condition of Auto-correction et oracle dynamique : certains effets n'apparaissent qu'à taille réduite on SemEval 2015 Task 18. There, the dynamic oracle raised semantic F1 from 0.870 to 0.891. The file has no reported GRAND-model scalar, so our replication supplies it.

Fairness rules apply throughout. Static and dynamic variants share hyperparameter search budgets, with the exploration temperature tuned on development data inside the same budget. Wall-clock time is reported because exploration adds sampling cost.

The key ablation separates the training signal from the exposure. In this condition the parser is trained on explored states but supervised only with the static oracle's next action wherever that action is still defined, and those states are skipped otherwise. This shows whether any reduction in $E$ comes from visiting corrupted states or from the dynamic oracle's correct targets there.

Beating the baselines is not the aim. The proposal succeeds if the decomposition predicts the held-out dynamic gains across cells better than parameter count does. Concretely, we fit on PTB/CTB and predict UD gains. It also succeeds if, at the large-encoder scale, dynamic training gives a statistically reliable reduction in mean $E$ even when in-domain LAS differences are within seed noise.

Why not just…?

Why not just compare static and dynamic accuracy across model sizes, as Zhao and Bernard did? That comparison cannot separate "makes fewer errors" from "recovers better," and the whole question turns on that difference. Two parsers with identical final accuracy can have very different recovery ability if one errs less often. We still report final accuracy in every cell, so a reader who prefers the simpler analysis gets it. The injection measure is what lets us say why the gap narrows.

Why not just use beam search at inference? Beam search hedges against local mistakes without teaching recovery, and it multiplies decoding cost by the beam width. Don’t Stop Me Now! Using Global Dynamic Oracles to Correct Training Biases of Transition-Based Dependency Parsers shows that beam training still has a train-test mismatch, with derivation endings under-represented, and that adding dynamic oracles to beam training gave up to 0.7 UAS on SPMRL. Beam search is still a real competitor at large scale. We include static-oracle beam decoding at small beam sizes as an extra condition and measure excess propagation under it. If beam search with a large encoder matches dynamic-oracle greedy parsing on the hard UD conditions at acceptable cost, we will say so.

Why not just train with sentence-level reinforcement learning? Tackling Error Propagation through Reinforcement Learning: A Case of Greedy Dependency Parsing found that plain REINFORCE was high-variance on long trajectories and was beaten by their oracle-informed policy gradient. Approximate Dynamic Oracle for Dependency Parsing with Reinforcement Learning used reinforcement learning to approximate an oracle rather than to replace it. Our question is about measurement, not a new training method, so reinforcement learning enters only as an optional extra row and not as the main comparison.

Why not accept that pretrained encoders already solve this? That is the hypothesis under test, and it could turn out true. Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle is weak counter-evidence: a 34.8M-parameter syntactic model trained with a dynamic oracle beat a 124.4M GPT-2 on syntactic generalization. That is a language-modeling result, though, not a parsing one. The injection experiment settles it. If mean $E$ under static training approaches zero for large encoders, recovery has been absorbed and dynamic oracles are unnecessary in that regime.

Decisive Checks

The data are English PTB Sections 02–21 for training and 23 for testing, CTB-5, the CoNLL 2017 UD treebanks (including the five used by de Lhoneux et al. and several small and non-projective ones), and SemEval 2015 Task 18 for the replication. Metrics are UAS and LAS, unlabeled exact match, mean excess propagation $E$ by error type and position, and the fraction of natural errors identified as propagated by recursive correction.

Three outcomes decide the question. First, for large pretrained encoders under static training, mean $E$ on matched injections is substantially above zero, and dynamic training lowers it. That would show recovery is not absorbed. Second, the stack-composition variant shows larger $E$ than the state-independent variant at equal parameter count. That would show the channel of state dependence matters more than size. Third, the dynamic gain on small and non-projective UD treebanks for large encoders rises with first-error rate as the decomposition predicts. Injected SWAP errors get special attention, because the static-dynamic oracle gives no recovery signal there. We expect excess propagation to stay high for those errors at every scale, which points to the place where a fully dynamic reordering oracle would pay off most.