Reducing Sequence Length by Predicting Edit Operations with Large Language Models

Masahiro Kaneko, Naoaki Okazaki

Abstract

Large Language Models (LLMs) have become the standard for sequence transduction tasks. However, for "local" editing tasks such as Grammatical Error Correction (GEC) or style transfer, where the majority of the source text remains unchanged, standard autoregressive generation is highly inefficient. This paper proposes a novel method that predicts edit spans—defined by their position in the source text and the corrected tokens—rather than regenerating the entire sequence. By shifting the model's objective to targeted editing, we reduce computational costs and mitigate the "copying bias" common in these tasks. Our experiments across four benchmarks (GEC, paraphrasing, style transfer, and simplification) demonstrate that this approach achieves state-of-the-art performance while significantly reducing target sequence length.

Introduction

When an LLM is tasked with correcting a single grammatical error in a long paragraph, it typically regenerates every token from scratch. This process is computationally expensive because Transformer models incur a cost that grows quadratically with the sequence length. Furthermore, since the input and output are nearly identical, models often develop a "copying bias," where they learn to repeat the input exactly to minimize training loss, thereby failing to make necessary corrections.

To address these issues, we propose a framework where the LLM identifies only the specific spans that require modification. This targeted approach ensures that computational resources are focused solely on the changes, effectively bypassing the redundancy of full-sentence regeneration.

Methodology

Edit Span Representation

The core of our method is representing edits as a set of coordinates. Instead of a full sentence, the model outputs a tuple: (start_position, end_position, new_tokens). For example, to change a verb tense at a specific index, the model identifies the numerical position and provides the replacement string.

The edit-span prediction pipeline: from source instruction to rule-based reconstruction.

Training and Reconstruction

To train the models, we utilized a linguistic alignment algorithm based on Damerau-Levenshtein distance to extract minimal edit sets from existing parallel corpora. We then performed instruction tuning on four open-source LLMs: MPT, OPT, LLaMA, and BLOOM. To maintain general reasoning capabilities, we employed a hybrid training strategy, mixing edit-span data with general-purpose instruction sets (Databricks-dolly-15k).

Post-inference, a simple rule-based script applies the predicted edit spans to the original source text to reconstruct the final, corrected output. This script is computationally negligible compared to the LLM inference itself.

Experimental Evaluation

We evaluated the proposed method on four distinct local sequence transduction tasks:

  • Grammatical Error Correction (GEC): NUCLE and CoNLL datasets.
  • Paraphrasing: Quora Question Pairs.
  • Formality Style Transfer: GYAFC corpus.
  • Text Simplification: WikiSmall, ASSET, and TurkCorpus.

Results and Discussion

Efficiency Gains

The primary advantage of our method is the reduction in sequence length. As shown in the experimental results, the position-based editing approach consistently produces much shorter target sequences than standard parallel generation. In tasks like GEC, the target sequence length was reduced by as much as 41% (and in some cases, the resulting output was only ~20% of the original length).

Comparison of target sequence lengths between standard parallel generation and position-based editing across different models and tasks.

Performance and Accuracy

Despite the reduced output length, the models did not sacrifice accuracy. In fact, task-specific fine-tuning with LLaMA achieved new state-of-the-art (SoTA) results across all four benchmarks. The models demonstrated a high degree of "coordinate awareness," successfully predicting correct numerical indices for word positions over 90% of the time after instruction tuning.

Generalization

Crucially, the hybrid training approach prevented the models from losing their general utility. When tested on the Dolly dataset, the models remained capable of handling open-ended prompts, proving that learning the edit-span format does not necessarily degrade broader linguistic intelligence.

Conclusion

By predicting edit spans instead of full sequences, we can make LLMs significantly more efficient for local transduction tasks. This method effectively eliminates copying bias and reduces the computational footprint of inference. While currently requiring fine-tuning to master numerical indexing, this approach provides a scalable path toward faster and more accurate automated text editing.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention. LLMs are also used for local sequence transduction tasks, including grammatical error correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged. However, the models that generate all target tokens in such tasks have a tendency to simply copy the input text as is, without making needed changes, because the difference between input and output texts is minimal in the training data. This is also inefficient because the computational cost grows quadratically with the target sequence length with Transformer. This paper proposes predicting edit spans for the source text for local sequence transduction tasks. Representing an edit span with a position of the source text and corrected tokens, we can reduce the length of the target sequence and the computational cost for inference. We apply instruction tuning for LLMs on the supervision data of edit spans. Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%. Furthermore, we report that the task-specific fine-tuning with the proposed method achieved state-of-the-art performance in the four tasks.
BibTeX
@inproceedings{kaneko2023reducing,
  author = {Masahiro Kaneko and Naoaki Okazaki},
  title = {{Reducing Sequence Length by Predicting Edit Operations with Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=OC4OLQGtIR},
}

Loading related content...

Figure