Traditional sequence-to-sequence (seq2seq) models generate text like a scribe: one word at a time, from left to right. While effective, this autoregressive process is inherently slow and prone to "hallucinations," where the model drifts away from the source facts. However, in many tasks—like grammatical error correction, sentence simplification, or dialogue rewriting—the output is mostly identical to the input.
A growing body of research suggests we should stop treating these tasks as "translation" and start treating them as editing. By predicting a set of parallel transformations (tags and insertions) rather than generating a new string from scratch, models can achieve massive speedups while remaining more faithful to the original text.
The Core Intuition: Tagging Instead of Typing
The fundamental shift in this research direction is moving from a generative decoder to a tagging mechanism. Instead of asking "What is the next word?", the model asks "What should I do with this current word?"
In the LASERTAGGER framework, Eric Malmi and colleagues demonstrate that most editing tasks can be reduced to three operations: KEEP, DELETE, or ADD_PHRASE. Because these tags are predicted for every input token simultaneously, the model bypasses the sequential bottleneck of autoregressive decoding. This parallel approach allows LASERTAGGER to run up to 100x faster than traditional models.
This "edit-first" philosophy provides a natural structural bias. As Yue Dong 0002 and Felix Stahlberg argue in their comprehensive overview, editing models are significantly less prone to hallucinations because they are physically constrained to modify the source rather than reinventing it.
Decomposing the Edit: Tagging and Insertion
While simple tagging works for deletions, adding new, context-specific content requires a more flexible approach. Modern architectures like FELIX, developed by Aliaksei Severyn and Jonathan Mallinson, decompose the problem into two distinct stages:
- Tagging: A non-autoregressive model marks tokens to be kept or deleted and identifies where new content is needed.
- Insertion: A separate mechanism fills in the blanks.
To handle complex reordering—a weakness of simple tagging—FELIX uses a pointer mechanism to "shuffle" the kept tokens before the insertion stage. This allows for global structural changes without losing the speed of parallel prediction. Similarly, Marine Carpuat and Sweta Agrawal show that training models with an editing curriculum that prioritizes these operations helps non-autoregressive models handle complex tasks like summarization.
Beyond Text: Editing as Vision and Segmentation
The "editing" paradigm is not limited to standard NLP. It has proven remarkably effective in specialized domains:
- Dialogue Rewriting: Qian Liu and Jian–Guang Lou reimagined incomplete utterance rewriting as a semantic segmentation task. By predicting an edit matrix over the words, they treat the sentence like an image where specific "pixels" (words) need to be modified.
- OCR and Handwriting: Instead of decoding characters one-by-one, LevOCR by Cheng Da uses a Levenshtein-based transformer to delete and insert characters in parallel. In a more radical shift, Raphaël Baena treats text recognition as parallel object detection, identifying all characters in a line at once.
- Image Captioning: Zhen Wang and Long Chen applied this to image caption editing, showing that refining an existing caption through parallel KEEP/DELETE/ADD operations is faster and more accurate than generating a new one from scratch.
Why It Works: The Efficiency of Parallelism
The primary driver behind this research is the massive reduction in inference latency. Standard LLMs are $O(N)$ in time complexity, where $N$ is the number of generated tokens. By contrast, parallel editing models like the LLaMA-based punctuation restorer by Debjyoti Paul achieve nearly 20x speedups by using a single forward pass to predict all edits.
Even when the process is iterative—as in EDITPRO by Graham Neubig and Machel Reid—the decomposition into "operation prediction" and "parallel span generation" ensures that the model remains efficient. By focusing on the delta between the input and output, these models leverage the high overlap inherent in editing tasks to do less work, more accurately, and much faster.