Abstract
Large Language Models (LLMs) have become the standard for sequence transduction tasks. However, for "local" editing tasks such as Grammatical Error Correction (GEC) or style transfer, where the majority of the source text remains unchanged, standard autoregressive generation is highly inefficient. This paper proposes a novel method that predicts edit spans—defined by their position in the source text and the corrected tokens—rather than regenerating the entire sequence. By shifting the model's objective to targeted editing, we reduce computational costs and mitigate the "copying bias" common in these tasks. Our experiments across four benchmarks (GEC, paraphrasing, style transfer, and simplification) demonstrate that this approach achieves state-of-the-art performance while significantly reducing target sequence length.
Introduction
When an LLM is tasked with correcting a single grammatical error in a long paragraph, it typically regenerates every token from scratch. This process is computationally expensive because Transformer models incur a cost that grows quadratically with the sequence length. Furthermore, since the input and output are nearly identical, models often develop a "copying bias," where they learn to repeat the input exactly to minimize training loss, thereby failing to make necessary corrections.
To address these issues, we propose a framework where the LLM identifies only the specific spans that require modification. This targeted approach ensures that computational resources are focused solely on the changes, effectively bypassing the redundancy of full-sentence regeneration.
Methodology
Edit Span Representation
The core of our method is representing edits as a set of coordinates. Instead of a full sentence, the model outputs a tuple: (start_position, end_position, new_tokens). For example, to change a verb tense at a specific index, the model identifies the numerical position and provides the replacement string.
Training and Reconstruction
To train the models, we utilized a linguistic alignment algorithm based on Damerau-Levenshtein distance to extract minimal edit sets from existing parallel corpora. We then performed instruction tuning on four open-source LLMs: MPT, OPT, LLaMA, and BLOOM. To maintain general reasoning capabilities, we employed a hybrid training strategy, mixing edit-span data with general-purpose instruction sets (Databricks-dolly-15k).
Post-inference, a simple rule-based script applies the predicted edit spans to the original source text to reconstruct the final, corrected output. This script is computationally negligible compared to the LLM inference itself.
Experimental Evaluation
We evaluated the proposed method on four distinct local sequence transduction tasks:
- Grammatical Error Correction (GEC): NUCLE and CoNLL datasets.
- Paraphrasing: Quora Question Pairs.
- Formality Style Transfer: GYAFC corpus.
- Text Simplification: WikiSmall, ASSET, and TurkCorpus.
Results and Discussion
Efficiency Gains
The primary advantage of our method is the reduction in sequence length. As shown in the experimental results, the position-based editing approach consistently produces much shorter target sequences than standard parallel generation. In tasks like GEC, the target sequence length was reduced by as much as 41% (and in some cases, the resulting output was only ~20% of the original length).
Performance and Accuracy
Despite the reduced output length, the models did not sacrifice accuracy. In fact, task-specific fine-tuning with LLaMA achieved new state-of-the-art (SoTA) results across all four benchmarks. The models demonstrated a high degree of "coordinate awareness," successfully predicting correct numerical indices for word positions over 90% of the time after instruction tuning.
Generalization
Crucially, the hybrid training approach prevented the models from losing their general utility. When tested on the Dolly dataset, the models remained capable of handling open-ended prompts, proving that learning the edit-span format does not necessarily degrade broader linguistic intelligence.
Conclusion
By predicting edit spans instead of full sequences, we can make LLMs significantly more efficient for local transduction tasks. This method effectively eliminates copying bias and reduces the computational footprint of inference. While currently requiring fine-tuning to master numerical indexing, this approach provides a scalable path toward faster and more accurate automated text editing.
Summary updated