Training Target-Language Taggers on Sets of Projected Labels, Not One Guess

Most named-entity recognizers for low-resource languages are built by borrowing labels from English. A tagger trained on English (or its predictions) is projected onto target-language sentences through machine translation, and a target-language tagger is then trained on these projected "silver" labels as if they were correct. This is a pipeline. Each stage commits to one answer, so a boundary shifted by one word or a dropped entity becomes a confident training error. We propose to stop committing. The projection step already produces several plausible labelings for each sentence. We keep them all and train the target tagger to put its probability on any of them. The tagger is a conditional random field, or CRF: a sequence-labeling layer that scores whole label sequences and rules out illegal ones, such as an "inside-person" tag with no "begin-person" before it. The CRF learns which hypothesis is right from regularities across the whole corpus.

The second move is where the silver labels live. We generate them on native target-language text, not translated text. Native sentences are translated into English, tagged there, and the predicted spans are projected back onto the original sentence. This avoids two problems. Models trained on translated text overfit to translation artifacts. And projection methods that depend on producing a clean target-language translation fail for languages the MT system handles poorly. The result is a single-pass tagger on native text. It needs no translation at test time and is trained from uncertain supervision without collapsing that uncertainty to one guess.

Core Claim

When projected labels are ambiguous, the target tagger should be trained with a set-marginal likelihood rather than a hard label. Concretely, let $C(x)$ be a small set of candidate label sequences for target sentence $x$, each with a projection-confidence weight $w(y)$. The tagger is trained to minimize

$$\mathcal{L}(x) = -\log \sum_{y \in C(x)} w(y)\, P_\theta(y \mid x),$$

where $P_\theta$ is the linear-chain CRF distribution over label sequences. We expect this to beat both training on the top-1 projection and filtering out low-confidence sentences, with the gain concentrated in sentences where projection hypotheses disagree about span boundaries.

The CRF is not decoration here. Because $C(x)$ is a set of whole sequences, the loss keeps the coupling between "where this span starts" and "where it ends." Token-wise soft labels throw that coupling away and can reward illegal mixtures of two hypotheses. The sum over $C(x)$ is computed exactly with a forward pass restricted to a small automaton, described below.

Multi-scale context enters as an extra constraint on $C(x)$, not as a new encoder. Token-level legality comes from the CRF transitions. Span-level alternatives come from the projection hypotheses. Corpus-level evidence comes from repeated surface strings: if a string is projected as an organization in most of its occurrences across the unlabeled corpus, candidates that type it differently are down-weighted. This corpus-level step is a hypothesis with its own ablation, since entity strings are sometimes genuinely ambiguous.

Relation to Prior Work

Three papers motivate the gap.

  • Constrained Decoding for Cross-lingual Label Projection (CODEC) first produces a clean translation, then constrains the MT decoder to insert entity brackets into it. It reranks top-$k$ bracketings and keeps one. We reuse its constrained decoding in the translate-test direction, where the fixed template is the native sentence itself. Unlike CODEC, we keep the top-$k$ hypotheses as a training set and distill them into a discriminative CRF tagger, rather than running two-pass MT at inference.
  • Multilingual Clinical NER: Translation or Cross-lingual Transfer? shows that translate-test suffers cascading errors for large encoders, yet beats direct transfer for small ones. Resilience to projection noise therefore depends on model capacity, so we test across encoder sizes.
  • Duh et al. show that training on translated text causes overfitting to translation artifacts. This motivates generating silver labels on native text.

Three papers supply method or constrain the design.

  • Low-Resource NER with Cross-Lingual, Character-Level Neural CRFs found that a CRF output layer and log-linear structure matter most when target data are tiny. We keep the CRF but replace their small gold target set with set-valued silver supervision.
  • Low Resource Multi-Task Sequence Tagging - Revisiting Dynamic CRFs down-weights noisy silver auxiliary labels by certainty inside a factorial CRF. Our weights $w(y)$ play the analogous role. The difference is that our uncertainty is over alternative labelings of the same task, not over a second task, so we avoid loopy inference.
  • A Practical Tool to Help Automate Interlinear Glossing documents how a hard CRF output fed into a dictionary lookup cascades errors. It is a reminder that 1-best handoffs are the failure point.

Two lines of work remove pipeline stages, and we contrast with both.

Our approach keeps the cheap pipeline and instead makes the downstream learner tolerant of the upstream ambiguity. The lattice models of Chinese NER Using Lattice LSTM and FLAT share the spirit of "keep the alternatives." They keep alternative segmentations at the input and need high-coverage lexicons. We keep alternative labelings at the output and need only an English tagger and target-to-English MT.

Research Program

Building the candidate sets. For each unlabeled native sentence, we translate it into English with NLLB-class MT and tag the English with a strong English NER model. We then project spans back in two ways: CODEC-style constrained decoding over the native sentence, which yields top-$k$ bracketings with reranker scores, and alignment-based projection.

Representing the set compactly. We align the hypotheses and split the sentence into regions where they agree and regions where they disagree. In each disagreement region, the region's options are its distinct labelings plus an "open" option. The open option allows any legal labeling, with a small weight, so that entities missed by every hypothesis are not forced to "outside." The candidate set is the product of region choices. This gives a small finite-state automaton, so the restricted forward pass costs about the same as ordinary CRF training.

Setting the weights. The weights $w(y)$ multiply the normalized reranker scores. They are optionally adjusted by the corpus-level type agreement of each candidate's entity strings.

Training the tagger. We fine-tune an XLM-R encoder with a CRF head on the loss above. Since the loss is a marginal likelihood, the CRF is free to resolve each set toward whichever hypothesis is most consistent with what it learns elsewhere. An optional second round re-scores candidate sets with the trained tagger, a constrained form of self-training.

Analysis. We will measure how often the trained tagger's Viterbi output, its single highest-scoring labeling, falls inside versus outside $C(x)$ on held-out sentences. We will also check whether its choices among candidates agree with gold labels more often than the reranker's top-1 choice does. That comparison is the direct test of whether corpus-wide learning disambiguates better than per-sentence scoring.

Baselines

Naive reference points. The first is zero-shot transfer: XLM-R plus a CRF fine-tuned on English CoNLL-2003 and applied directly to each target language. The second is the stronger zero-shot recipe from MasakhaNER 2.0. It transfers from a LangRank-chosen African source language and reports an average gain of 14 F1 over English sources, 85.5 F1 for isiXhosa to isiZulu, and roughly the value of 500 annotated target sentences. Both are run with the same encoder and tuning budget as our method.

Strongest existing method. This is CODEC applied as a translate-test system at inference. It reports an average F1 of 70.4 across 18 MasakhaNER 2.0 languages, against 65.8 for alignment-based projection, and +16.5 F1 over EasyProject on chiShona (Constrained Decoding for Cross-lingual Label Projection). We will reproduce it with the same MT model and English tagger that we use to build our candidate sets. The comparison then isolates what happens after projection.

Beating CODEC. Beating it means exceeding its reproduced average on the same 18 languages without any MT at test time. If our reproduction falls short of 70.4, we report both numbers. A tie at a fraction of the inference cost would also be a useful result, but we would report it as a tie.

Ablations that isolate the key ingredient. All use the same candidate generation, encoder, data, and number of tuning trials.

  • Hard-label silver CRF: the same CRF trained on CODEC's top-1 projection only. This is the closest competitor, and beating it is the main claim.
  • Confidence filtering: the hard-label CRF trained only on sentences whose top-1 reranker margin exceeds a threshold tuned on a small development set.
  • Token-wise soft labels: the hypotheses' votes collapsed into per-token label distributions, used to train a softmax head and, separately, a CRF.
  • Set-CRF without the corpus-consistency weights: this isolates the multi-scale component.

As an upper reference we train on 100 and 500 gold target sentences.

Why not just…?

Why not just keep CODEC's top-1 projection and train a normal tagger? This is the hard-label ablation, and it is the most important comparison. Our expectation rests on CODEC's own finding that its reranked top-$k$ hypotheses differ in quality and that top-1 is imperfect. Committing to it reproduces the 1-best handoff that Domain adaptation in practice and the glossing study identify as the source of cascading collapse. If the set-CRF gains less than about one F1 on average and nothing on high-disagreement sentences, the idea is wrong, and we would say so.

Why not just discard uncertain sentences? Filtering is a real threat. With plenty of unlabeled text, dropping ambiguous sentences may cost little. However, ambiguous sentences are disproportionately those with multi-word or morphologically complex entities, exactly the cases the tagger most needs to see. We test this by reporting F1 broken down by entity length and by language morphology type, with the filtering baseline tuned fairly. If filtering matches us everywhere, the practical recommendation becomes "filter."

Why not just use per-token soft labels with a softmax head? It is simpler and needs no automaton. Per-token marginals discard which start goes with which end. Averaging two hypotheses that disagree on a boundary can therefore reward a labeling that neither hypothesis proposed. The token-soft-label ablations, run with both softmax and CRF heads, measure whether this coupling matters in practice. We also count illegal or fragmented spans in each system's output, the failure Structured prediction models for RNN based sequence labeling in clinical text attributes to independent token decisions.

Why not just do zero-shot transfer from the best related language? MasakhaNER 2.0 shows that this is strong where a close relative with annotations exists. It is our second naive baseline. We also train the set-CRF on top of that model, so the two can combine. The silver-data approach should matter most for languages without a well-resourced close relative, and we will report results split by that condition.

Proposed Experiments and Decisive Checks

Setup. The benchmark is MasakhaNER 2.0, using the unlabeled training-split sentences of each of the 20 languages as native text and the official test sets for evaluation. The metric is entity-level F1. Encoders are distilled mBERT, XLM-R base and XLM-R large, to probe the capacity effect reported for clinical NER. Every number is reported over three seeds.

First decisive check. Does the set-CRF beat the hard-label silver CRF on average, with the gap growing as candidate-set disagreement grows?

Second decisive check. Does a test-time-MT-free tagger reach or exceed reproduced CODEC translate-test F1 on the 18-language subset?

Third decisive check. Does the corpus-consistency weighting add accuracy, or does it hurt on languages where entity strings are ambiguous?

Robustness check. We degrade the MT system by using a smaller model. Hard-label pipelines should lose more than the set-based learner if the method really absorbs upstream error.

Out-of-domain probe. We apply the same recipe to the German clinical set from Multilingual Clinical NER. This tests whether the method holds outside news text. That paper reports no exact F1 values in the material we have, so we will reproduce its translate-test pipeline as the reference.