EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints

Weijia Xu, Marine Carpuat

This paper, from researchers at the University of Maryland, introduces EDITOR, a non-autoregressive (NAR) Transformer model designed to make Neural Machine Translation (NMT) both faster and more flexible. By introducing a novel "reposition" operation, the authors decouple the choice of which words to use from where they should be placed, allowing for seamless integration of user-provided lexical constraints.

TL;DR

EDITOR is a non-autoregressive translation model that uses a "reposition" operation to move words around during iterative refinement. This allows it to incorporate user-specified terminology (lexical constraints) more effectively and up to 33% faster than previous state-of-the-art edit-based models.

What Problem Does This Paper Address?

Standard NMT models are "autoregressive," meaning they generate text one word at a time from left to right. While accurate, this makes it incredibly difficult to force the model to use specific words (lexical constraints) without using "constrained beam search," which is computationally expensive and slow.

Non-autoregressive (NAR) models were invented to speed things up by generating words in parallel. However, existing NAR models like the Levenshtein Transformer (LevT) only use "insertion" and "deletion" operations. If a word is in the wrong place, LevT has to delete it and then re-insert it elsewhere. This "entangles" word choice with positioning, leading to inefficiencies and making it harder to satisfy user constraints (like "always translate gleznă as ankle") without producing clunky, disfluent sentences.

Methods

The core innovation is the EDITOR (Edit-Based Transformer with Repositioning). Instead of just adding or removing words, EDITOR can move them.

  1. The Reposition Operation: For every word in a draft translation, the model predicts a new position (or decides to delete it). This allows the model to reorder a "bag of words" into a coherent sentence in a single step.
  2. Iterative Refinement: Translation happens in cycles: Reposition $\rightarrow$ Placeholder Insertion $\rightarrow$ Token Prediction. The model starts with an initial guess (which can be the user's constraints) and refines it until it's satisfied.
  3. Dual-Path Imitation Learning: To train the model, the authors use a "dual-path" strategy. The repositioning model and the insertion model are trained to fix each other's mistakes. This mimics the iterative process used during actual translation (inference).
  4. Soft Constraints: Unlike "hard" constraints that force a word into the output regardless of grammar, EDITOR treats user-provided words as "soft" constraints. It starts the translation process with these words already in the sequence, then uses its repositioning and insertion powers to build a fluent sentence around them.

Datasets

The authors evaluated EDITOR on three standard machine translation benchmarks:

  • WMT16 Romanian-English (Ro-En)
  • WMT14 English-German (En-De)
  • WAT2017 English-Japanese (En-Ja)

To specifically test lexical constraints, they used terminology sets extracted from Wiktionary and IATE (Interactive Terminology for Europe), which provide specific domain-specific terms that must be correctly translated.

Key Insights and Ideas

  • Disentangling "What" from "Where": By allowing the model to move words, EDITOR separates the lexical choice (selecting the right words) from the reordering task. This is a return to classical MT concepts (like the IBM models) but modernized for the Transformer era.
  • Efficiency through Movement: A single repositioning move can replace a long sequence of deletions and insertions. This makes the model faster, especially for language pairs like English-Japanese that require significant reordering.
  • The Power of "Soft" Constraints: The paper shows that starting with a bag of constraint words and letting the model "reposition" them leads to much higher fluency than forcing words in as "hard" constraints, which often causes grammatical errors.
  • Better for Rare Words: EDITOR significantly outperforms previous NAR models on low-to-medium frequency words, likely because the repositioning operation allows it to better utilize the specific vocabulary it is given.

Key Results

  • Speed: EDITOR is 6–7% faster than the Levenshtein Transformer on Ro-En and En-De, and a massive 33% faster on English-Japanese.
  • Quality: It achieves comparable or better BLEU scores (a measure of translation quality) than LevT across all tasks.
  • Constraint Satisfaction: On terminology tasks, EDITOR preserved significantly more user-provided constraints (7–17% more) than LevT while maintaining higher fluency.
  • Comparison to Autoregressive Models: EDITOR achieves quality similar to standard autoregressive Transformers but with a 2x to 4x speedup in decoding.

Limitations and Future Work

  • Token-Level Only: EDITOR operates on tokens. If a word is moved to a position where it needs a different grammatical inflection (e.g., changing "run" to "runs"), the model must rely on its insertion/deletion mechanism to fix the suffix, rather than "mutating" the word directly.
  • Complex Constraints: The current work focuses on simple word-level constraints. The authors suggest that future work could explore more complex constraint types, such as stylistic or structural preferences.
  • Beyond Translation: The "repositioning" logic is general-purpose. The authors propose applying EDITOR to other sequence generation tasks like text summarization or conversational AI.

Summary updated

Abstract

We introduce an Edi t-Based T ransf O rmer with R epositioning (EDITOR), which makes sequence generation flexible by seamlessly allowing users to specify preferences in output lexical choice. Building on recent models for non-autoregressive sequence generation (Gu et al., 2019), EDITOR generates new sequences by iteratively editing hypotheses. It relies on a novel reposition operation designed to disentangle lexical choice from word positioning decisions, while enabling efficient oracles for imitation learning and parallel edits at decoding time. Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer (Gu et al., 2019) while speeding up decoding dramatically compared to constrained beam search (Post and Vilar, 2018). EDITOR also achieves comparable or better translation quality with faster decoding speed than the Levenshtein Transformer on standard Romanian-English, English-German, and English-Japanese machine translation tasks.
BibTeX
@article{xu2021editor,
  author = {Weijia Xu and Marine Carpuat},
  title = {{EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=FvGOsN02EcO},
}

Loading related content...

Figure