Tree Ensembles Over Neural Scores for Rare Translation-Error Detection

Word-level quality estimation asks a model to mark which words in a machine translation are wrong, without a reference translation to compare against. It has one dominant difficulty: most words are fine. A tagger that labels every word "OK" looks accurate and is useless. The standard fix is to fine-tune a multilingual encoder such as XLM-RoBERTa with a linear output layer, reweight the loss, and hand-tune a probability cutoff. Even the winning WMT 2023 system, NJUNLP, describes those cutoffs as a fragile manual step. We propose to leave the neural tagger as it is and add a second stage. A gradient-boosted or balanced-bagged tree ensemble would re-score each target word and each gap between words. It would combine the tagger's own output with signals the tagger never sees directly: disagreement across tagger seeds, word-alignment gaps to the source, the translation system's token probabilities, length and number/entity mismatches, and sentence-level context. The ensemble's vote margin would then set the cutoff for calling an error.

The bet is specific. The rare errors a fine-tuned encoder misses, especially omissions and severe mistakes, are not random. They correlate with cheap, heterogeneous, mostly language-agnostic signals. Tree ensembles are the tool with the best track record for mixing such signals, and they have already worked on skewed translation problems. So a small tree stage should improve the ranking of rare errors, and therefore precision-recall and Matthews correlation on the minority class, beyond what more loss reweighting or more neural ensembling achieves. If the gain disappears once a logistic-regression stacker gets the same features, or once the tagger is simply averaged over seeds, the claim is false, and the experiments are designed to show that.

What exists and what is different

Three strands of prior work meet here. The first is FBK-UPV-UEdin's WMT14 system (FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task). It fed a word-level recurrent tagger's predictions, along with 79 hand-built features, into Extremely Randomized Trees (forests whose split points are chosen at random) to predict sentence-level post-editing effort. That paper motivates the stacking design. However, it used trees only at the sentence level and left word-level imbalance to the recurrent tagger, whose over-prediction of "OK" it explicitly reports. We move the tree stage to the word and gap level, which is where the imbalance lives.

The second is a random-forest lexicon-induction study (Using a Random Forest Classifier to Compile Bilingual Dictionaries of Technical Terms from Comparable Corpora). There, millions of wrong candidate pairs outnumber the correct ones. Ranking candidates by the forest's vote margin instead of a hard vote raised top-1 accuracy from 0.11 to 0.41, and kept roughly 59% top-20 accuracy on rare terms where context vectors fell toward zero. We reuse that margin-as-ranking idea and apply it to error tokens.

The third is DOFEN, which makes oblivious trees differentiable and matches XGBoost and CatBoost on tabular benchmarks. It also reports that plain deep networks still trail boosted trees on heterogeneous tabular features. We read that as support for putting a tree learner on the heterogeneous feature layer. We deliberately do not follow DOFEN's route of a differentiable tree head trained end to end on 1024-dimensional encoder states. The research file flags that such heads are slow to train and run. More importantly, a differentiable forest head sitting on the same encoder states as the linear layer sees no information the linear layer lacks. We expect it to behave like an MLP head, so we include it as an ablation rather than as the method.

On the error-detection side, the constraints are clear. Loss reweighting did not rescue rare adequacy categories such as "Unrelated" and "Contradiction" (Is This Translation Error Critical?). Omissions are the hardest errors because nothing in the output marks them (Findings of the WMT 2022 Shared Task on Quality Estimation). Levenshtein Training for Word-level Quality Estimation addresses gaps with insertion heads and reaches MCC 0.477 on WMT20 English-German against OpenKiwi's 0.358. Our gap-level stage targets the same weakness with alignment-derived features rather than a new decoder.

Core Claim

A tree ensemble trained on out-of-fold neural tagger outputs plus heterogeneous alignment, uncertainty and consistency features will improve the ranking of minority-class errors (BAD words, omission gaps, Major/Critical spans) over the tagger alone. The improvement should be concentrated on tokens where the tagger is uncertain and on omissions. Two ingredients are separable: the added features, and the tree learner's ability to combine them non-linearly under imbalance. We will measure each separately. A third claim is secondary: balanced bagging with an out-of-bag vote margin gives a decision cutoff that is more stable than a cutoff tuned on a small dev set. This matters most for low-resource pairs.

Research Program

The first stage is a standard XLM-RoBERTa-Large word and gap tagger with class-weighted cross-entropy. It is trained in $K$-fold cross-fitting, so every training token receives a prediction from a model that never saw it. Without this, the tree stage would learn from overconfident in-sample scores.

For every target word and every gap, we then build a feature row. The row contains the tagger's BAD probability, averaged over five seeds, and the spread of that probability across seeds. It contains the entropy of the tagger's prediction and of its neighbours'. It contains the source words aligned to the target word, or for a gap the count of source words left unaligned. For gaps, this unaligned-source count is the main omission signal. The row also includes the translation model's log-probability of the token, where the translation system is available. MLQE-PE translations came from released fairseq Transformer models (MLQE-PE), which makes this feasible for that corpus. The remaining features are number, named-entity and negation mismatch flags, relative position, sentence length ratio, and the predicted sentence-level score.

We fit two tree learners on these rows: LightGBM-style gradient boosting, and a balanced random forest in which every bootstrap sample keeps all errors and subsamples OK tokens. For the forest, the ranking score of a token $x$ is the margin

$$m(x) = \frac{1}{B}\sum_{b=1}^{B}\big(\mathbb{1}[h_b(x)=\text{BAD}] - \mathbb{1}[h_b(x)=\text{OK}]\big),$$

where $h_b$ is the $b$-th of $B$ trees. The cutoff on $m$ is chosen to maximize MCC on out-of-bag votes, meaning each training token is scored only by the trees that did not see it. This removes the need for a separate dev set to set the cutoff. For MQM severity spans, the same stage is trained as an ordinal problem and adjacent tokens are merged into spans. This replaces the hand-set $\epsilon_{minor}, \epsilon_{major}$ cutoffs of NJUNLP.

Baselines

The naive reference is the constant "all OK" predictor, together with the untuned first-stage tagger thresholded at 0.5. Together they show how much of any gain is simply calibration. On MLQE-PE, the published Predictor-Estimator baseline reports an average F1-BAD of 0.579 (MLQE-PE), which anchors our reimplementation.

The strongest practical competitor is the same tagger averaged over five seeds, with class-weighted loss and an MCC-optimal threshold tuned on the full dev set. It uses the same backbone, data and GPU budget as our first stage. It is what a careful practitioner would do, and it already captures part of what seed disagreement gives us.

For reported state of the art, the strongest word-level number in the file for WMT20 English-German is MCC 0.477 from Levenshtein Training. For WMT23 English-German, NJUNLP reports word-level MCC 29.7 and span F1 28.4 (NJUNLP). We will quote these but not claim to beat them unless we stack on a comparably strong first stage. The fair test is whether our second stage adds to whichever first stage we use, including a pseudo-data-trained one. xCOMET is a span-level MQM baseline and is run as released. The file contains no reported span F1 for it on these test sets, so we measure that number ourselves.

Three ablations isolate the key ingredient. The first is a logistic-regression stacker and a two-layer MLP stacker on exactly the same feature rows, with a matched hyperparameter search of 50 trials each. This separates "trees" from "features." The second is the tree stage given only the tagger's own outputs, which separates "features" from "second stage." The third is a DOFEN-style differentiable forest head on encoder states, which tests whether tree structure helps without new information.

Beating the strongest competitor means three things. On held-out test sets, the gain is at least two MCC points and a larger gain in area under the precision-recall curve for BAD tokens. It is significant under a paired bootstrap over sentences. It holds on a majority of language pairs.

Why not just…?

Why not just reweight the loss or use focal loss and tune the threshold? Tuned reweighting is inside our strongest baseline, so the comparison is direct. The existing evidence says it is not enough on the tail: sample-wise loss scaling left "Unrelated" and "Contradiction" poorly predicted (Is This Translation Error Critical?). Reweighting changes how much each error counts but gives the model no new evidence about omissions. If reweighting plus threshold tuning matches our method on precision-recall area, that finding would itself show that imbalance in this task is only a calibration problem, and we would report it.

Why not just generate more synthetic errors? Pseudo-data is the real threat. It is what won WMT 2023. Yet NJUNLP still needed manual cutoffs and still saw severity skew. WMT 2022 critical-error detection dropped sharply when systems could not use error-type-specific synthetic data (WMT 2022 Findings), which suggests synthetic data helps most on the error types it imitates. Our method is complementary, not competing. We run the second stage on top of a pseudo-data-trained first stage. If the added gain vanishes there, the tree stage is only compensating for a weak tagger, and we will say so.

Why not just ensemble more neural taggers, or add a bigger MLP head? Seed averaging is our main baseline, and the MLP stacker on identical features is an ablation. DOFEN's tabular results suggest trees combine heterogeneous features better than MLPs (DOFEN), but that evidence comes from tabular data, not from this feature layer. The MLP-versus-tree comparison decides the question here. If the MLP matches the trees, the contribution is the features, and the paper's claim narrows accordingly.

Proposed Experiments

We run word- and gap-level tagging on MLQE-PE across all eleven pairs, on WMT20 English-German for comparison with Levenshtein training, and on the WMT22 MQM English-German, English-Russian and Chinese-English data and WMT23 English-German for severity spans. Metrics are MCC and F1-BAD, area under the precision-recall curve for the minority class, and recall at a fixed precision of 0.5, which reflects post-editing triage. On MQM data we also report recall broken down by error category and severity, with omissions separated out.

The first decisive check is whether gains appear on gap (omission) tags and on the tagger's uncertain decile. If they appear only on tokens the tagger already gets right, the method does nothing useful.

The second check addresses the cutoff claim. On low-resource pairs (Nepali-, Sinhala-, Pashto- and Khmer-English), we subsample the dev set to 100, 300 and all sentences. We compare the variance of test MCC under dev-tuned thresholds against the out-of-bag margin threshold. The margin threshold should be less variable at small dev sizes and should never be much worse at full size.

The third check tests transfer. We train the tree stage on high- and medium-resource pairs and apply it unchanged to held-out low-resource pairs, over the same first-stage tagger. Most of the features do not depend on the language, so this measures whether the second stage carries its gains across language pairs where rare-error data is scarcest.