Abstract
Large Language Models (LLMs) have become the standard for various natural language processing tasks. However, in local sequence transduction tasks—such as grammatical error correction (GEC) and style transfer—LLMs often exhibit inefficiencies. Because these tasks involve minimal changes to the source text, models typically generate the entire target sequence from scratch, leading to redundant computation and a "copy bias" where the model fails to make necessary edits. This paper proposes a novel approach: training LLMs to predict only "edit spans"—specific tuples containing the position and the corrected tokens. Our experiments across four tasks (GEC, paraphrasing, style transfer, and simplification) demonstrate that this method reduces target sequence length by up to 32% while achieving state-of-the-art performance.
Introduction
Local sequence transduction refers to tasks where the input and output sequences share a high degree of overlap. In a typical workflow, an LLM receives a sentence and is instructed to rewrite it (e.g., "Make this sentence more formal"). Despite only a few words needing adjustment, standard autoregressive models generate every token of the output. This is problematic for two reasons:
- Computational Cost: The self-attention mechanism in Transformers scales quadratically with sequence length. Generating unchanged tokens is a waste of resources.
- The Copying Trap: Training data for these tasks is often dominated by identical source-target tokens, causing models to over-learn the identity mapping and ignore the editing instructions.
Methodology: Edit Span Prediction
To address these issues, we shift the model's output space from full-text generation to a set of discrete edits.
Edit Span Extraction and Representation
Using a linguistic alignment algorithm, we identify differences between source and target texts. These differences are represented as tuples: (start_index, end_index, "corrected_tokens"). For example, to change "scholars are affected" to "scholars have been affected," the model would output a tuple identifying the index of "are" and the replacement string "have been."
Instruction Tuning
We fine-tune several open-source LLMs (including MPT, OPT, LLaMA, and BLOOM) using a mixture of edit-span data and general instruction-following data (Databricks-Dolly). The models are prompted to output a comma-separated list of edit tuples. A rule-based post-processing script then reconstructs the final sentence by applying these edits to the original source text.
Experimental Results
We evaluated the proposed method on four benchmarks: NUCLE/CoNLL (GEC), Quora Question Pairs (Paraphrasing), GYAFC (Style Transfer), and WikiSmall/ASSET (Simplification).
Sequence Compression
The most immediate benefit of our approach is the reduction in output length. On average, the target sequence length was reduced by 32% across all tasks. In GEC specifically, where edits are often very sparse, we saw a 21% reduction. This significantly lowers the number of decoding steps required during inference.
Performance and State-of-the-Art
As shown in the table below, the "Edit spans" approach consistently matches or exceeds the performance of the "Plain" full-text baseline. When combined with task-specific fine-tuning on LLaMA, our method achieved new state-of-the-art results across all four evaluated tasks.
Discussion and Limitations
While highly effective for local edits, this method has limitations. It is less applicable to tasks like translation or creative summarization where the entire sentence structure changes. Furthermore, we observed a "zero-shot gap": high-end models like GPT-4 struggle to generate accurate indices without specific fine-tuning on the edit-span format. Future work will focus on making the reconstruction process more robust to minor indexing errors and exploring few-shot capabilities for index learning.
Conclusion
By reframing local sequence transduction as an edit-span prediction task, we can significantly improve the efficiency and accuracy of LLMs. This approach mitigates copy bias by forcing the model to focus exclusively on required changes, providing a faster and more reliable path for automated text editing.
Summary updated