Teaching Entity-Masked Translators to Survive Their Own Extraction Pipeline

When a machine translation system must handle specialized text such as biomedical abstracts, it usually cannot learn the rare technical terms from parallel data, because that data barely exists. The common fix is to find the terms in the source sentence with an entity detector, look up their translations in a dictionary, and hand those translations to the translator. One way to do the handoff is to replace each term with a placeholder slot. Masking a term this way, sometimes called entity masking or blinding, also helps with scarce data: the translator learns the sentence frame around a slot once and reuses it for thousands of terms it has never seen. Every published version of this idea, however, assumes the detector and dictionary are right. In deployment they are not. A clipped span, a spurious detection, or a wrong dictionary sense becomes a corrupted slot, and the translator copies it faithfully into the output. This is the classic cascading failure of pipelines.

We propose training the masking translator on the specific mistakes its own upstream pipeline makes. The plan has three parts. First, we measure how the deployed detector and dictionary fail on a small annotated development set. Second, we corrupt the masked training inputs by sampling from that measured failure profile rather than from uniform noise. Third, we attach the detector's confidence to every slot and keep the original source words visible beside the placeholder, so the model can learn when to trust a slot and when to translate around it. The claim is that this recovers most of the constraint adherence of hard placeholders on correct constraints, refuses most incorrect ones, and helps most when in-domain parallel data is smallest.

Core Claim and Position Relative to Prior Work

The work closest to ours is Understanding and Improving the Robustness of Terminology Constraints in Neural Machine Translation. It shows that pure placeholders reach 94.4% sentence-level constraint accuracy but damage fluency. Its fix, Robust Terminology Translation (RTT), has the model predict a term label, then append the actual term text, and it keeps the source term next to the label. RTT supplies our base architecture. It is evaluated only with oracle constraints, so it never learns to reject a slot. That gap is exactly what we target.

Two papers show that soft handling resists noise. Neural Machine Translation With Noisy Lexical Constraints treats constraints as attention memory the decoder may ignore. When 3 of 5 constraints are wrong, it holds 37.74 BLEU on Chinese-English, while hard grid beam search falls from 42.17 to 33.02. Integrating Vectorized Lexical Constraints for Neural Machine Translation encodes constraints as attention keys and values. These papers motivate soft trust, but neither masks the source, so neither gets the slot-reuse benefit under scarcity. Their noise is also synthetic or absent, not measured from a real extractor.

Extract and Attend: Improving Entity Translation in Neural Machine Translation is the closest system to a real pipeline. It runs entity detection, dictionary lookup, and transliteration, then feeds the candidate translations to the decoder as a prefix. Its authors concede that misidentified entities "may mislead" the model, and they do not train against that failure.

The general lesson about soft evidence comes from Relieving the Computational Bottleneck: Joint Inference for Event Extraction with High-Dimensional Features. It passes upstream classifier margins as soft evidence instead of hard labels. We carry that lesson into a neural translator cheaply, through a per-slot confidence input, rather than through costly joint inference.

Two further papers constrain the design. Training Neural Machine Translation to Apply Terminology Constraints shows that a model trained on inline terms learns to copy them and can inflect them. That is good for morphology but dangerous with a noisy dictionary, since the learned copying bias is precisely what propagates errors. RTT's results also show that inline code-switching collapses to 71.9% accuracy at high term density. Together these rule out pure code-switching for dense biomedical text.

Research Program

The method has one formal object: the pipeline's error profile. Let $c$ be a gold constraint, meaning a source span paired with its target term. Let $\hat c$ be what the deployed detector-plus-dictionary emits for it. Let $s$ be the detector's confidence. On a few hundred annotated in-domain sentences we estimate

$$q(\hat c, s \mid c) ,$$

the joint distribution over error type and confidence. The error types are exact, boundary-clipped, boundary-extended, wrong dictionary sense, missed, and spurious detection with no gold counterpart.

At training time, each gold constraint obtained by matching a termbase against both sides of the bitext is replaced by a draw from $q$. Every slot then carries three things: its placeholder label, the original source words (as in RTT's source enhancement), and a binned version of $s$ as an extra input embedding.

The decoder's target sequence follows RTT, with one change. When the sampled constraint is wrong, the reference contains the correct translation instead of the label-plus-term pattern. The model is therefore rewarded for overriding a bad slot from source context. On correct slots, the loss mask over the internal tokens of the term is kept as RTT specifies.

The mechanism tells us when masking should help. A slot is useful when the sentence frame around it is the scarce thing to learn. It is harmful when the model needs the term's meaning to translate around it, which is why the source words stay visible. We will vary three things: in-domain bitext size, constraint density, and the upstream noise rate. The noise rate is varied by swapping detectors of different quality and by injecting additional errors drawn from the measured profile.

Experimental Setting

The main setting is English-Spanish biomedical translation, using the WMT19 Biomedical Translation Shared Task data described in Findings of the WMT 2019 Biomedical Translation Shared Task. It has a useful terminology sub-task: 6,624 English terms manually translated into Spanish, on which the top system reached only 47.55% accuracy. We use part of this list as the dictionary. A held-out part tests generalization to unseen terms.

To model data scarcity, we subsample the in-domain bitext to roughly 10k and 50k sentence pairs and compare against the full set. Training follows the general-to-health-to-biomedical transfer schedule of UCAM Biomedical Translation at WMT19.

A secondary setting is the English-German high-density benchmark from the RTT paper, which averages 6.1 constraints per sentence. We run it twice, once with its oracle constraints and once with constraints produced by a real detector, to show what oracle evaluation hides.

The metrics are BLEU, COMET, and sentence-level constraint accuracy. We add two pipeline-aware measures: correct-term recall on slots that are right, and wrong-term copy rate, the fraction of incorrect slots whose term surfaces in the output.

Baselines

The naive reference point is the unconstrained domain-adapted Transformer trained with the Saunders et al. transfer schedule. That paper reports 41.3 BLEU on the Scielo Bio validation set with the Health→All-biomed→Bio schedule, against 36.1 for training on Bio alone, in UCAM Biomedical Translation at WMT19. We rerun it on our subsampled splits so the comparison is fair. Our method must beat it on term accuracy without losing more than a fraction of a COMET point.

The strongest masking competitor is RTT Understanding and Improving the Robustness of Terminology Constraints in Neural Machine Translation. Its reported figures are 91.9% sentence-level constraint accuracy, 40.2 BLEU, and 0.4866 COMET on the English-German high-density benchmark with oracle constraints. We train RTT on the same data, with the same architecture size and the same number of tuning runs, and feed it the same noisy pipeline constraints at test time. Beating it means matching its accuracy within one point on correct slots while cutting wrong-term copy rate substantially.

The strongest noise-robust competitor is the soft-memory model SE-ATTN Neural Machine Translation With Noisy Lexical Constraints. It reports 37.74 BLEU on Chinese-English with 3 of 5 constraints noisy, versus 33.02 for grid beam search. We also include Dinu et al.'s Replace-with-source-factors method Training Neural Machine Translation to Apply Terminology Constraints, which reports 93–94% term usage, and the Extract-and-Attend decoder prefix Extract and Attend. Every system receives identical upstream constraints and an equal hyperparameter search budget.

Three ablations isolate the key ingredient. The first replaces the measured error profile with uniform corruption at the same overall rate. The second removes the confidence input. The third removes masking entirely but keeps profile-matched noisy soft constraints. Only the full model should win in the low-data, high-density corner.

Why not just…?

Why not just fine-tune on the in-domain data and skip constraints entirely? At 10k sentence pairs most test terms never occur in training. The WMT19 terminology result of 47.55% for the best system shows that even well-resourced shared-task systems mistranslate isolated technical terms. The unconstrained baseline row measures exactly this. If it matches our term accuracy at full data size, we would report that masking pays off only under scarcity, which is itself a useful finding.

Why not just use soft constraint memory, as SE-ATTN does, and drop the masking? This is the real threat. Soft memory already resists noise, and it may also resist pipeline noise. Our bet is that soft memory alone gives no slot-reuse benefit, so its advantage should shrink as bitext shrinks and density rises. Placeholder-style labels, by contrast, held 94.4% accuracy where code-switching fell to 71.9% in the RTT study. The no-masking ablation settles the question. If it ties the full model across data sizes, the masking component is unjustified and we would drop it.

Why not just train with random noise in the constraints? Real extractor errors are structured. Boundary clipping on multiword drug names, for instance, is far more common than random substitution. A model trained on uniform noise may learn to distrust every slot equally, which lowers recall on correct ones. The uniform-noise ablation tests this directly. If uniform noise matches the measured profile, the method simplifies, and we would say so.

Why not just collapse the pipeline into one joint model? Collapsing stages into a single tagger works for extraction tasks, as in Biomedical Event Extraction as Sequence Labeling. For translation, though, the dictionary is an external resource that changes after training, and a joint model cannot absorb new terms without retraining. Our design keeps the dictionary swappable, which the held-out-term test checks.

Decisive Checks

The idea succeeds if three things hold. Wrong-term copy rate must fall well below that of RTT and Dinu et al. at equal correct-term recall. The gap must widen as in-domain bitext shrinks from full size to 10k. The measured-profile training must beat uniform noise on the real-detector test.

The idea fails if SE-ATTN without masking matches it in the low-data, high-density setting, or if oracle-constraint rankings on the RTT benchmark predict real-pipeline rankings perfectly. In that second case, measuring the pipeline's errors would add nothing.