EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints

Weijia Xu, Marine Carpuat

Abstract

We introduce an Edit-Based Transformer with Repositioning (EDITOR), which makes sequence generation flexible by seamlessly allowing users to specify preferences in output lexical choice. Building on recent models for non-autoregressive sequence generation, EDITOR generates new sequences by iteratively editing hypotheses. It relies on a novel reposition operation designed to disentangle lexical choice from word positioning decisions, while enabling efficient oracles for imitation learning and parallel edits at decoding time. Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer (LevT) while speeding up decoding dramatically compared to constrained beam search. EDITOR also achieves comparable or better translation quality with faster decoding speed than the Levenshtein Transformer on standard Romanian-English, English-German, and English-Japanese machine translation tasks.

Introduction: The Challenge of Lexical Constraints

Standard Neural Machine Translation (NMT) models are typically autoregressive (AR), generating text sequentially from left to right. While effective, this approach makes incorporating specific user-provided words—lexical constraints—computationally expensive, often requiring slow "constrained beam search" techniques.

Non-autoregressive (NAR) models were developed to improve speed by generating tokens in parallel. However, existing edit-based NAR models like the Levenshtein Transformer (LevT) rely solely on insertion and deletion. If a word is generated in the wrong position, the model must delete and re-insert it, a process that entangles word choice with positioning. This inefficiency makes it difficult to satisfy user constraints without compromising fluency.

The EDITOR Model

EDITOR addresses these limitations by introducing a novel reposition operation. This allows the model to decouple the selection of tokens from their placement in the sequence.

The Reposition Operation

Instead of just adding or removing words, EDITOR can move them. For every word in a draft translation, the model predicts a new position or decides to delete it. This allows the model to reorder a "bag of words" into a coherent sentence in a single step, significantly reducing the number of iterations required for refinement.

The reposition operation $\mathcal{E}(y, r)$ allows EDITOR to reorder tokens in a single step, disentangling lexical choice from positioning.

Iterative Refinement and Soft Constraints

Translation in EDITOR occurs in cycles: Reposition $\rightarrow$ Placeholder Insertion $\rightarrow$ Token Prediction. The model starts with an initial hypothesis—which can include user-specified constraints—and refines it until convergence.

By treating user-provided words as "soft" constraints, EDITOR starts the process with these words already in the sequence and then uses its repositioning and insertion capabilities to build a fluent sentence around them. This approach avoids the grammatical errors often seen when forcing "hard" constraints into a fixed structure.

Experimental Results

The authors evaluated EDITOR on three standard benchmarks: WMT16 Romanian-English (Ro-En), WMT14 English-German (En-De), and WAT2017 English-Japanese (En-Ja).

Performance and Speed

EDITOR achieves translation quality (BLEU scores) comparable to or better than the Levenshtein Transformer while offering significant speed improvements. Notably, EDITOR is 6–7% faster than LevT on Ro-En and En-De, and approximately 33% faster on English-Japanese, a language pair requiring substantial reordering.

Performance comparison across three translation benchmarks. EDITOR matches or exceeds the quality of the Levenshtein Transformer (LevT) while offering lower latency.

Constraint Satisfaction

In terminology-constrained tasks using Wiktionary and IATE datasets, EDITOR preserved 7–17% more user-provided constraints than LevT. It also showed marked improvements in handling low-to-medium frequency words, as the repositioning operation allows the model to better utilize the specific vocabulary it is given.

Conclusion

EDITOR represents a significant advancement in non-autoregressive translation by modernizing classical MT concepts of reordering for the Transformer era. By disentangling "what" to say from "where" to say it, the model provides a faster, more flexible framework for NMT that excels at satisfying lexical constraints without sacrificing fluency. Future work may extend this repositioning logic to other sequence generation tasks such as summarization or conversational AI.

Summary updated

Abstract

We introduce an Edit-Based Transformer with Repositioning (EDITOR), which makes sequence generation flexible by seamlessly allowing users to specify preferences in output lexical choice. Building on recent models for non-autoregressive sequence generation (Gu et al., 2019), EDITOR generates new sequences by iteratively editing hypotheses. It relies on a novel reposition operation designed to disentangle lexical choice from word positioning decisions, while enabling efficient oracles for imitation learning and parallel edits at decoding time. Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer (Gu et al., 2019) while speeding up decoding dramatically compared to constrained beam search (Post and Vilar, 2018). EDITOR also achieves comparable or better translation quality with faster decoding speed than the Levenshtein Transformer on standard Romanian-English, English-German, and English-Japanese machine translation tasks.
BibTeX
@article{xu2021editor,
  author = {Weijia Xu and Marine Carpuat},
  title = {{EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=FvGOsN02EcO},
}
Figure