RLDE: A Reinforcement Learning Framework for Interpretable Iterative Document Editing

Nikhil Parkar, M. Marimuthu, Riya Joshi, Karthikeyan NarXiv 2026 · arXiv

TL;DR

RLDE (Reinforcement Learning based Document Editing) is a novel framework that treats document revision as an iterative, step-by-step decision process rather than a single-pass rewrite. By combining a hierarchical "Manager-Worker" architecture with a reference-free reward system, RLDE achieves high-quality, interpretable edits that rival large language models (LLMs) like GPT-4 at a fraction of the computational cost.

What Problem Does This Paper Address?

Human writers don't just vomit out a perfect final draft; they refine text through multiple rounds of revisions. However, most automated editing systems struggle to replicate this. Current approaches generally fall into three buckets, each with a major flaw:

  1. Rule-based systems: Inflexible and unable to handle high-level features like flow or readability.
  2. Supervised models: Require massive amounts of expensive, paired "before-and-after" data.
  3. Large Language Models (LLMs): Act as "black boxes" that rewrite entire blocks of text at once, often hallucinating or losing the original author's intent.

Furthermore, when Reinforcement Learning (RL) has been applied to text in the past, it often suffered from "catastrophic deletion"—the agent learns that the easiest way to improve a readability score is to simply delete most of the document.

Table 1 Comparative Analysis of Text Editing Paradigms, Controllable RL Frameworks, and RLDE

Methods

The authors propose RLDE, which formulates editing as a Markov Decision Process (MDP). The system is built on a Coarse-to-Fine Hierarchical Architecture:

  • The Manager: A deterministic scheduler that looks at the whole document and identifies which paragraphs have the highest "linguistic deficit" (poor readability or high error density).
  • The Worker: A PPO (Proximal Policy Optimization) agent that operates on the chosen paragraph. It chooses from four discrete actions: NOOP (do nothing), INSERT, DELETE, or REPLACE.

Key Technical Innovations: * Decoupled Control and Generation: The RL agent doesn't generate text directly. Instead, it uses a Pointer Network to select a specific span of text and then calls a frozen BART-base model to suggest a context-aware replacement or insertion. This makes the process highly interpretable—you can see exactly which action was taken and why. * Reference-Free Multi-Objective Reward: The agent is trained to optimize a combination of readability (Flesch Reading Ease), fluency (Perplexity), and grammar scores. * Semantic Preservation Penalty: To stop the agent from "cheating" by deleting text, the authors introduced a penalty based on SBERT cosine similarity. If the edited version drifts too far in meaning from the original, the agent is penalized.

Datasets

The primary dataset used for training and evaluation is the BEA-2019 ABCN Dataset, which contains a mix of native and non-native English essays.

Table 2 Corpus Statistics for the BEA-2019 ABCN Dataset

To test generalizability, the authors also performed zero-shot evaluations on:

  • JFLEG: Focused on fluency and style.
  • CoNLL-2014: A standard benchmark for Grammatical Error Correction (GEC).
  • WikiLarge: Used for text simplification tasks.

Key Insights and Ideas

  • Interpretability through Primitives: By restricting the agent to discrete actions (INSERT, DELETE, etc.), the system provides a clear "audit trail" of revisions, mimicking human editorial logs.
  • Solving the Deletion Trap: The paper proves that hard semantic constraints are essential for RL in NLP. Without the SBERT penalty, semantic retention dropped to 62%; with it, it stayed at 93%.
  • Efficiency is King: RLDE uses a compact architecture (~163M total parameters, with only 54K being trainable policy parameters). This allows it to run on a standard CPU with an inference latency of just 45ms per paragraph.
  • Reference-Free Training: Because the reward is based on intrinsic quality metrics rather than matching a "gold" human edit, the model can theoretically learn to improve text for which no perfect version exists.

Key Results

RLDE showed significant improvements across the board on the BEA-2019 test set:

  • Readability: Flesch Reading Ease increased from 49.8 to 59.4.
  • Fluency: Perplexity (a measure of how "natural" the text is) decreased from 63.7 to 39.8.
  • Human Validation: In double-blind evaluations, RLDE scored 4.9/5 for meaning preservation and 4.4/5 for overall edit quality, outperforming supervised baselines like GECToR.
  • Zero-Shot Performance: On the JFLEG dataset, RLDE (48.7 GLEU) significantly outperformed LLaMA-2 (46.5 GLEU) and approached the performance of GPT-4 (51.3 GLEU), despite being orders of magnitude smaller and faster.

Limitations and Future Work

While RLDE is highly effective at improving readability and fluency, the authors noted that its grammar correction gains were more modest compared to specialized supervised models.

Future directions include: * Integrating the Pointer Network and the generative model into a single, unified transformer framework. * Incorporating explicit discourse models (like Rhetorical Structure Theory) to improve high-level document coherence. * Developing adaptive "budgets" that allow the agent to spend more time on complex paragraphs and less on simple ones.

Summary updated

Abstract

Improving the quality, clarity, and flow of long documents remains a challenging problem in NLP, as human editing typically occurs through iterative refinement rather than a single-pass rewrite. Most existing automated systems act as black-box generators or rely on supervised learning, limiting interpretability and failing to capture the sequential nature of revisions. In this paper, we propose RLDE, an automated end-to-end metric-driven RL framework that formulates document revision as a reference-free MDP problem (unsupervised with respect to paired document-revision targets). Working with a limited action space (NOOP, INSERT, DELETE, REPLACE), RLDE leverages Pointer Networks for dynamic span selection at the token level and sends generative actions to the frozen BART-base model to obtain context-aware suggestions. PPO agents learn to optimize a multi-objective reward function based on readability, perplexity, and grammar metrics with the help of a novel SBERT-based Semantic Preservation Penalty constraint to avoid deletion degeneracy. Evaluations on the BEA-2019 dataset showed statistically significant results (p < 0.01) across all metrics, where Flesch Reading Ease increased from 49.8 to 59.4, perplexity decreased from 63.7 to 39.8, while preserving 93% of semantic information. When compared to Chain-of-Thought (CoT) prompt-engineered models like GPT-4 and LLaMA-2, RLDE shows high performance at a very low computational inference cost. Moreover, the quality of edits generated by RLDE has been verified via human evaluations, scoring 4.9/5 for meaning preservation and 4.4/5 for edit quality.
BibTeX
@misc{parkar2026rlde,
  author = {Nikhil Parkar and M. Marimuthu and Riya Joshi and Karthikeyan N},
  title = {{RLDE: A Reinforcement Learning Framework for Interpretable Iterative Document Editing}},
  year = {2026},
  eprint = {2611.22039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22039},
}
Figure