Combining several parsers used to be one of the most reliable ways to improve syntactic parsing. Here a parser means a model that maps a sentence to a dependency tree, a set of head-to-word links. The gain came from combining parsers that made different mistakes, such as a fast left-to-right "transition-based" parser and a "graph-based" parser that scores every possible link at once. That source of improvement appears to have dried up. Once both parsers read the same deep contextual word representations, their errors become about as similar as two random seeds of the same model, and blending them stops helping (Integrating Graph-Based and Transition-Based Dependency Parsers in the Deep Contextualized Era). That study used ELMo and an older parser design. Nobody has asked the natural follow-up question: does diversity come back if ensemble members are built on encoders with different pretraining histories? And if it does, is the gain worth paying for once the ensemble's extra compute is given to a single, larger parser instead?
We propose a controlled study, run through one locked evaluation harness, that answers both questions. We separate four sources of diversity: random seed, decoding paradigm, encoder pretraining, and training-data view. For each, we measure how correlated the members' arc errors are and how much a tree-valid ensemble gains. Every comparison is matched for inference compute and tuning budget, and every result is reported both in domain and under domain shift. In two sentences: modern parser ensembles may have failed because their members shared a representation, not because ensembling stopped working. We test this in a way that cannot be explained away by tuning effort or by the ensemble simply being bigger.
Core Claim
Our hypothesis is that heterogeneous pretraining restores the arc-level error diversity that shared contextual encoders removed. Under a fixed inference budget, this diversity should yield ensemble gains that a single larger parser does not match. We further expect three things. The gains should appear mostly out of domain and on long-distance and non-projective links. The gain should be predictable in advance from a single measured quantity, the excess error correlation between members. And most of the gain should survive distillation of consensus trees into one student parser.
The excess error correlation is defined over the arcs $a$ of a held-out set. Let $e_i(a)\in\{0,1\}$ indicate that member $i$ attaches the word to the wrong head. Then
$$\rho_{ij} = \mathrm{corr}_a\big(e_i(a), e_j(a)\big) - \bar\rho_{\text{seed}},$$
where $\bar\rho_{\text{seed}}$ is the mean correlation between two seeds of the same configuration. A value of $\rho_{ij}$ near zero means the pair is no more diverse than a re-run. The falsifiable prediction is that ensemble gain in Labeled Attachment Score (LAS) increases with $-\rho_{ij}$. LAS is the fraction of words assigned both the correct head and the correct relation label.
The idea differs from the closest work in specific ways. Falenska et al. varied only the decoding paradigm under a shared ELMo encoder, and they did not match ensembles against larger single models. Ensemble Models for Dependency Parsing: Cheap and Good? established, for pre-neural parsers, that member diversity matters far more than the cleverness of the combination rule. We test whether that conclusion transfers when diversity has to be bought through pretraining. Ensemble Distillation for Unsupervised Constituency Parsing showed that heterogeneous architectures overlap less than seeds and that distilling an explicit "average tree" avoids over-smoothed students. That evidence motivates us, but it comes from unsupervised constituency parsing. We supply the supervised dependency counterpart.
Research Program
The first component is the harness, which carries the standardized-evaluation lesson. BARS found that properly tuned old recommenders matched new ones. A Sober Look at the Unsupervised Learning of Disentangled Representations traced reported gains to tuning and seed choice. Both warn that ensemble gains can be manufactured the same way.
The harness fixes the following:
- gold tokenization and sentence segmentation, because the CoNLL 2018 shared task could not separate ensembling effects from preprocessing pipelines;
- one evaluator for LAS and UAS (Unlabeled Attachment Score, which ignores labels);
- an identical hyperparameter search budget per member and per single-model baseline;
- five seeds per configuration;
- paired significance tests per treebank.
All configurations and logs are released.
The second component is the member pool. The base is the biaffine graph-based parser (Deep Biaffine Attention for Neural Dependency Parsing), which scores every head–dependent pair with a bilinear function. Diversity is varied one axis at a time:
- Seed: same everything else.
- Decoding paradigm: biaffine versus a transition-based decoder on the same encoder. This repeats Falenska et al. with Transformers.
- Encoder pretraining: for English, RoBERTa versus ELECTRA, following the heterogeneous-pretraining ensemble in the DialDoc21 report. For the multilingual UD treebanks, XLM-R versus another multilingual encoder.
- Data view: members trained on different bootstrap resamples.
The third component is combination. Members vote on labeled arcs, either with equal weights or with their arc probabilities. The highest-scoring valid tree is then recovered by maximum spanning tree decoding (Chu-Liu-Edmonds), as in The CoNLL 2007 Shared Task on Dependency Parsing. We also run the linear-time repair that Surdeanu and Manning found equal to exact decoding.
The fourth component is a distilled student. A single biaffine parser is trained on consensus trees decoded from the ensemble over unlabeled in-domain text. This applies the average-tree idea to dependencies, and it answers the inference-cost objection directly.
Baselines
Naive reference: one tuned parser. A single biaffine parser on one base-size encoder, tuned with the shared budget. Its seed-to-seed spread defines the noise floor every claimed gain must exceed. On PTB the published biaffine figures are 95.7% UAS and 94.1% LAS (Deep Biaffine Attention for Neural Dependency Parsing). For Transformer-backbone biaffine parsers on the UD v2.4 treebanks or OntoNotes, we found no reported result on Lacuna, so we reproduce these numbers inside the harness.
Strongest competitor: compute-matched single parser. A single biaffine parser whose encoder is large enough to match the ensemble's total inference compute, for example one large encoder against two or three base ones. It gets the same data and the same total tuning budget. This is the alternative a skeptical reviewer would actually deploy. Beating it means a positive LAS difference that is significant on a majority of treebanks, with confidence intervals excluding zero on the out-of-domain averages.
Same-backbone ensembles. A seed ensemble and a paradigm ensemble (graph-based plus transition-based) that share one encoder, combined with the identical decoder. This reproduces the regime where Falenska et al. found no significant blending gain. The comparison between these ensembles and the heterogeneous-pretraining ensemble isolates our key ingredient. If the heterogeneous ensemble does not beat the seed ensemble, the core claim fails.
Training-time sharing. Graph/transition multi-task sharing of one encoder, which Falenska et al. report improves the transition parser by 0.42 LAS on average at no runtime cost. Alongside it, the DCST-ENS structural self-training ensemble serves as the zero-overhead competitor in the low-resource and domain-adaptation settings.
Pre-neural historical anchor. The cheap voting ensemble of Surdeanu and Manning, with 88.83% LAS in domain and 81.99% out of domain on CoNLL 2008. We use this only to compare the size of the in-domain versus out-of-domain gap, since the treebanks differ.
Why not just…?
Why not just use one bigger parser? This is the real threat. Biaffine parsing reached strong results with no ensembling at all, and the BARS experience suggests that tuning a simple model often closes apparent gaps. Our answer is the compute-matched baseline itself. If the large single parser ties the heterogeneous ensemble both in domain and on OntoNotes domain shift, we report that ensembling is not worth its cost for supervised dependency parsing. That would be a useful negative result, because it would extend Falenska et al. to Transformers. Our bet on a gain under domain shift rests on the out-of-domain ensemble gains in Surdeanu and Manning and in the SUSANNE results of the ensemble-distillation paper. This is evidence from other settings, not a guarantee.
Why not just ensemble random seeds or checkpoints? Seed and checkpoint ensembles are cheap and need no second pretrained model. If they recover most of the gain, pretraining diversity is beside the point. The seed-ensemble baseline and the $\rho_{ij}$ measurement settle this directly. If heterogeneous pairs show excess correlation near zero, the hypothesis is wrong.
Why not just vote arc by arc and repair cycles greedily? Surdeanu and Manning found that unweighted voting matched learned weighting, and that greedy linear-time repair matched exact decoding. We expect this to hold here as well, so we do not present a sophisticated combiner as a contribution. Both decoders are run. If they tie, we adopt the cheaper one, and the study's claim rests on member diversity alone.
Why not just distill or share an encoder during training? Multi-task sharing gives a free 0.42 LAS. However, it shares one encoder, which is exactly what we suspect erodes diversity. The distilled-student arm tests the stronger version of this objection: if a single student trained on consensus trees keeps most of the ensemble's gain, the deployment argument against ensembles disappears, and the recommendation becomes "ensemble at training time only."
Proposed Experiments
Datasets. The main testbed is the 13 UD v2.4 treebanks used by Falenska et al. They span non-projectivity rates and word orders and provide direct comparability with the diversity-collapse finding. The domain-shift testbed is English OntoNotes 5.0, training on one genre and testing on the other six, following the DCST setup. We also include its 500-sentence lightly supervised condition, where diversity may matter most. PTB serves as a sanity check against the published biaffine figures.
Metrics. We report LAS and UAS overall, by dependency length, and on non-projective arcs, along with $\rho_{ij}$ for every member pair. We also report arc-level calibration of vote agreement, meaning whether the fraction of members agreeing on an arc tracks its probability of being correct. This last measure tests whether the ensemble provides usable confidence, which point metrics in shared tasks ignore.
Decisive checks. There are four:
- Whether the heterogeneous-pretraining ensemble exceeds both the seed ensemble and the compute-matched single parser, with the margin larger out of domain than in domain.
- Whether, across all member pairs and treebanks, ensemble gain is predicted by $-\rho_{ij}$. A regression slope reliably above zero would turn the result into a cheap screening rule: measure correlation on a small dev set before paying for an ensemble.
- Whether the paradigm-only ensemble reproduces Falenska et al.'s null result under Transformers, which validates the harness against a known outcome.
- What fraction of the ensemble-over-single gain the distilled student retains at single-model inference cost.
If the first check fails and the third succeeds, the conclusion is clean: shared pretraining is not the bottleneck, and supervised parser ensembles have run their course.
References
- Integrating Graph-Based and Transition-Based Dependency Parsers in the Deep Contextualized Era
- Ensemble Models for Dependency Parsing: Cheap and Good?
- Ensemble Distillation for Unsupervised Constituency Parsing
- BARS: Towards Open Benchmarking for Recommender Systems
- A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
- CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies
- Deep Biaffine Attention for Neural Dependency Parsing
- Technical Report on Shared Task in DialDoc21
- The CoNLL 2007 Shared Task on Dependency Parsing
- Deep Contextualized Self-training for Low Resource Dependency Parsing