When a model trained on English is applied to Hindi, Korean or Japanese, part of what goes wrong is word order. English puts the verb before its object, and these languages put it at the end. A long-standing fix is pre-ordering: parse the English sentence, rearrange its phrases into target-language order, and train on the rearranged text. The fix still helps modern multilingual models. It also has a known weakness. It commits to a single rearrangement even when the parser was wrong, or when the target language allows several orders. We propose to predict, for every phrase under a shared head word, a continuous position score together with a calibrated interval around it, built with conformal prediction (a method that turns held-out errors into intervals with guaranteed coverage). Sorting the scores gives an order. Overlapping intervals mark places where several orders are plausible. Wide intervals caused by unfamiliar constructions mark places where the system should not trust itself.
The practical payoff is a reordering policy driven by two separated sources of doubt. When the target language is genuinely flexible, the system trains on several sampled orders. When the system simply does not know, because the construction is rare or the parse is shaky, it leaves the English order alone. Two kinds of prior pre-ordering systems each do part of this: hard solver-based systems commit to one order, and lattice systems hedge by heuristic. Neither tells these two cases apart. Continuous uncertainty methods have the tools to separate them but have only been tested on images, molecules and tables.
Core Claim
Word order inside a dependency subtree can be written as a small continuous regression problem. Each dependent of a head receives a target position score in $[0,1]$, its normalized rank among its siblings in the target language. We claim three things about this formulation. First, it supports conformal guarantees at the level of whole subtrees. Second, it admits a principled split of uncertainty into a noise floor and a model-ignorance term. Third, using that split to decide when to reorder, when to sample, and when to abstain improves downstream cross-lingual transfer over committing to one order, over always mixing original and reordered text, and over sampling orders from raw corpus statistics.
What is new here is not that order is "really continuous". The scalar score is a device that makes interval-based uncertainty and coverage guarantees available. The contribution is a calibrated decision rule for pre-ordering whose value can be tested.
Prior Work and Its Role
Three pre-ordering systems motivate and constrain the design.
Improving Cross-lingual Transfer through Subtree-aware Word Reordering does three things. It estimates pairwise ordering probabilities between dependency labels from target-language Universal Dependencies treebanks. It turns these into hard constraints for a constraint solver. It then reorders English subtrees recursively. Even with XLM-R and mT5 it reports gains of 2.3 to 4.1 labeled attachment score (LAS) points on distant languages. This paper is our strongest baseline and also our evidence that the word-order gap survives pretraining. It also discards the probabilities at exactly the point where they could inform a hedge. We keep them, and we calibrate them.
Fuzzy Syntactic Reordering for Phrase-based Statistical Machine Translation showed that forcing parser-based reorderings can hurt. It also showed that a lattice offering optional reordering or fuzzified subject boundaries helps, recovering about 69% of the gain obtainable with gold parses. That result motivates abstention when the parse is doubtful. However, its alternatives are unweighted heuristics. The same paper warns that dense lattices crowd out correct paths, which argues for small, calibrated sets rather than broad hedging.
Word Reordering for Zero-shot Cross-lingual Structured Prediction learns soft adjacency predictions inside an order-insensitive encoder and improves LAS on 21 of 30 languages. It shows that soft order signals are usable, but it provides no confidence or coverage of any kind.
From uncertainty quantification we take four methods, each for a specific job. Split conformal regression supplies the coverage machinery. Density-ratio weighting, as in CoDrug, handles the fact that English subtrees are not distributed like target subtrees. The decomposition of risk under the continuous ranked probability score (CRPS) into Bayes risk and excess risk, from Uncertainty Quantification for Regression using Proper Scoring Rules, supplies the split between "the language is flexible" and "the model is ignorant". That paper also reports that excess risk is the better signal for out-of-distribution detection. Finally, How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts? constrains the design. There, no regression uncertainty method held 90% coverage under real shift, and a Gaussian ensemble fell to about 59% on Cells-Tails. We therefore treat unweighted conformal intervals as a baseline expected to fail, not as the method.
Research Program
The predictor takes a head word and its dependents. It sees dependency labels, part-of-speech tags, subtree sizes, and contextual multilingual embeddings of the head and dependents. It outputs, for each dependent, a two-component Gaussian mixture over its position score. We use a mixture because flexible languages produce genuinely bimodal placements, such as a scrambled object before or after the subject. Training data come only from target-language UD treebanks. For every target head, we compute each dependent's normalized sibling rank and train with CRPS, which needs no parallel text. A small deep ensemble of five members supplies the excess-risk estimate. The Bayes-risk estimate comes from the mixture's own spread, following the Fishkov et al. decomposition.
For calibration we hold out target-language subtrees. We define one nonconformity score per subtree, the largest normalized residual among its dependents:
$$s = \max_{j} \frac{|r_j - \hat{\mu}_j|}{\hat{\sigma}_j}$$
Here $r_j$ is the observed position score of dependent $j$, and $\hat{\mu}_j, \hat{\sigma}_j$ are the predicted mixture mean and spread. Taking the $(1-\alpha)$ quantile of $s$ gives intervals whose joint coverage holds for the whole subtree, not just for each dependent. The set of admissible orders is then every permutation consistent with the intervals.
When we apply the model to English subtrees, we reweight the calibration scores by an estimated density ratio between English and target subtree features. The ratio comes from a classifier trained to tell English subtrees from target ones. This is the CoDrug recipe moved from molecular features to subtree features.
The reordering policy is recursive and top-down, and it moves whole subtrees as units, as in Arviv et al. At each head, the system abstains and keeps English order if either the excess risk or the parser's arc-marginal entropy exceeds a threshold. Otherwise, if the admissible set contains one order, it uses that order. If the set contains several, it samples from them in proportion to the mixture probabilities when producing training copies. All thresholds and the $\alpha$ level are tuned on target-language development data for the downstream task. Every compared system gets the same tuning budget.
Baselines
The naive reference is training on English in its original order, with no reordering, on the same XLM-R and mT5 backbones. Every gain is reported relative to it.
The strongest existing method is the subtree-aware solver reordering of Arviv et al., run in its ensemble form that trains on both original and reordered English. Its reported figures are LAS gains of 2.3 to 4.1 points on typologically distant UD v2.10 languages, exact-match gains of 2.0 to 6.1 points on MTOP and Multilingual TOP, and relation-classification F1 gains of up to 10.4 points in ensemble settings. Absolute scores are not in the file, so no reported result found on Lacuna for the absolute LAS of this system. We will rerun it ourselves.
For fairness, every system receives the same English training sentences, the same target treebank sentences for statistics, the same parser, and the same backbone. Every system is also capped at the same number of training copies per sentence: two for the ensemble comparison and four for the sampling comparison. This matters because our method's sampling would otherwise get more data. Beating this baseline means a statistically significant improvement, over three seeds, in average LAS across the distant languages. The exact-match and F1 averages must not get worse.
Two ablations isolate the key ingredient. The first is POD sampling: draw orders directly from Arviv et al.'s pairwise ordering probabilities without intervals, abstention or calibration, under the same copy budget. The second is our system with abstention driven only by parser arc-marginal entropy, with no excess-risk term. We will also run unweighted conformal intervals and a temperature-scaled ensemble to test whether the shift correction matters. We include a lattice-style optional-reordering variant, in which each subtree is reordered or not with probability one half, as the Andreas et al. analogue.
Why not just…?
Why not just let mT5 or XLM-R learn reordering end to end? Arviv et al. ran exactly these models and still found the typological gap, with encoder-decoder models more sensitive to it. The no-reordering baseline above reproduces this check in our setup.
Why not just sample orders from the pairwise ordering statistics, with no uncertainty model? This is the most serious threat. Corpus statistics already encode much of the target's flexibility. If POD sampling under a matched copy budget equals our system, then the continuous intervals add nothing, and we will report that. The method can only win in two places. One is where calibration sharpens the sample set, which the Andreas et al. lattice-crowding result suggests matters. The other is where abstention avoids reordering misparsed sentences. Both effects are measured separately.
Why not just gate reordering on the parser's own confidence? Parser marginals capture parse doubt but not unfamiliar target constructions. They also say nothing about target flexibility. This is a real competitor and is included as an ablation. If it matches the excess-risk gate, we will drop the ensemble and keep the simpler gate.
Why not just use an ensemble or temperature scaling for confidence? Gustafsson et al. found ensembles lose coverage badly under shift. English-to-target application is a shift by construction. The intrinsic coverage experiment below tests whether this holds for subtrees.
Proposed Experiments
The intrinsic test asks whether the intervals are honest. On held-out target-language subtrees from a different genre than the calibration data, we measure joint subtree coverage at a nominal 90%, along with average admissible-set size and Kendall's tau between predicted and observed orders. We compare unweighted conformal, weighted conformal, and the ensemble. We expect unweighted intervals to under-cover and weighted ones to come close to 90%. We add one check with a real source language: Arabic-to-English order on the LDC2007E103 parallel text used by Andreas et al., with ground-truth order taken from automatic word alignments. We report coverage there and compare against their 64% correct verb-subject ordering.
The downstream test uses the UD v2.10 nine-language parsing suite, including Arabic, Hindi, Korean and Thai, measured by LAS. It also uses MTOP and Multilingual TOP for exact match, and TACRED translations and IndoRE for F1. All systems from the Baselines section are run under matched budgets, both with full target treebanks and with the 1,000-sentence regime Arviv et al. report as sufficient.
The decisive checks are three. First, our system must beat POD sampling at a matched copy budget. Second, its gains should concentrate in languages with freer order, such as Hindi, Korean and Japanese, since that is where the Bayes-risk branch acts. The typologically close Italian condition should show no loss. Third, when we degrade the parser, by training it on fewer sentences or applying it to out-of-domain English, the abstaining system should degrade more slowly than solver reordering. If none of these three effects appears, the claim that calibrated continuous uncertainty helps pre-ordering is false.
References
- Improving Cross-lingual Transfer through Subtree-aware Word Reordering
- Fuzzy Syntactic Reordering for Phrase-based Statistical Machine Translation
- Word Reordering for Zero-shot Cross-lingual Structured Prediction
- CoDrug: Conformal Drug Property Prediction with Density Estimation under Covariate Shift
- Uncertainty Quantification for Regression Using Proper Scoring Rules
- How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?