Reducing Sequence Length by Predicting Edit Spans with Large Language Models

Masahiro Kaneko, Naoaki Okazaki

Abstract

Large Language Models (LLMs) have become the standard for various natural language processing tasks. However, in local sequence transduction tasks—such as grammatical error correction (GEC) and style transfer—LLMs often exhibit inefficiencies. Because these tasks involve minimal changes to the source text, models typically generate the entire target sequence from scratch, leading to redundant computation and a "copy bias" where the model fails to make necessary edits. This paper proposes a novel approach: training LLMs to predict only "edit spans"—specific tuples containing the position and the corrected tokens. Our experiments across four tasks (GEC, paraphrasing, style transfer, and simplification) demonstrate that this method reduces target sequence length by up to 32% while achieving state-of-the-art performance.

Introduction

Local sequence transduction refers to tasks where the input and output sequences share a high degree of overlap. In a typical workflow, an LLM receives a sentence and is instructed to rewrite it (e.g., "Make this sentence more formal"). Despite only a few words needing adjustment, standard autoregressive models generate every token of the output. This is problematic for two reasons:

  1. Computational Cost: The self-attention mechanism in Transformers scales quadratically with sequence length. Generating unchanged tokens is a waste of resources.
  2. The Copying Trap: Training data for these tasks is often dominated by identical source-target tokens, causing models to over-learn the identity mapping and ignore the editing instructions.

Methodology: Edit Span Prediction

To address these issues, we shift the model's output space from full-text generation to a set of discrete edits.

Edit Span Extraction and Representation

Using a linguistic alignment algorithm, we identify differences between source and target texts. These differences are represented as tuples: (start_index, end_index, "corrected_tokens"). For example, to change "scholars are affected" to "scholars have been affected," the model would output a tuple identifying the index of "are" and the replacement string "have been."

Figure 1: The proposed workflow where an LLM predicts edit spans that are then applied to the source text via rule-based conversion.

Instruction Tuning

We fine-tune several open-source LLMs (including MPT, OPT, LLaMA, and BLOOM) using a mixture of edit-span data and general instruction-following data (Databricks-Dolly). The models are prompted to output a comma-separated list of edit tuples. A rule-based post-processing script then reconstructs the final sentence by applying these edits to the original source text.

Experimental Results

We evaluated the proposed method on four benchmarks: NUCLE/CoNLL (GEC), Quora Question Pairs (Paraphrasing), GYAFC (Style Transfer), and WikiSmall/ASSET (Simplification).

Sequence Compression

The most immediate benefit of our approach is the reduction in output length. On average, the target sequence length was reduced by 32% across all tasks. In GEC specifically, where edits are often very sparse, we saw a 21% reduction. This significantly lowers the number of decoding steps required during inference.

Performance and State-of-the-Art

As shown in the table below, the "Edit spans" approach consistently matches or exceeds the performance of the "Plain" full-text baseline. When combined with task-specific fine-tuning on LLaMA, our method achieved new state-of-the-art results across all four evaluated tasks.

Table 1: Performance comparison between standard full-text generation (Plain) and the proposed edit span method across different LLM backbones.

Discussion and Limitations

While highly effective for local edits, this method has limitations. It is less applicable to tasks like translation or creative summarization where the entire sentence structure changes. Furthermore, we observed a "zero-shot gap": high-end models like GPT-4 struggle to generate accurate indices without specific fine-tuning on the edit-span format. Future work will focus on making the reconstruction process more robust to minor indexing errors and exploring few-shot capabilities for index learning.

Conclusion

By reframing local sequence transduction as an edit-span prediction task, we can significantly improve the efficiency and accuracy of LLMs. This approach mitigates copy bias by forcing the model to focus exclusively on required changes, providing a faster and more reliable path for automated text editing.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention. LLMs are also used for local sequence transduction tasks, including grammatical error correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged. However, the models that generate all target tokens in such tasks have a tendency to simply copy the input text as is, without making needed changes, because the difference between input and output texts is minimal in the training data. This is also inefficient because the computational cost grows quadratically with the target sequence length with Transformer. This paper proposes predicting edit spans for the source text for local sequence transduction tasks. Representing an edit span with a position of the source text and corrected tokens, we can reduce the length of the target sequence and the computational cost for inference. We apply instruction tuning for LLMs on the supervision data of edit spans. Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%. Furthermore, we report that the task-specific fine-tuning with the proposed method achieved state-of-the-art performance in the four tasks.
BibTeX
@inproceedings{kaneko2023reducing,
  author = {Masahiro Kaneko and Naoaki Okazaki},
  title = {{Reducing Sequence Length by Predicting Edit Spans with Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=OC4OLQGtIR},
}
Figure