Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revision

Wanyu Du, Zae Myung Kim, Vipul Raheja, Dhruv Kumar, Dongyeop Kang

TL;DR

This paper introduces R3, a novel human-in-the-loop system for iterative text revision that allows AI models to collaborate with human writers. R3 aims to improve the efficiency and quality of text revision by enabling continuous human-machine interaction, where users accept or reject AI-generated edits, which then inform subsequent revisions.

What Problem Does This Paper Address?

Writing is an inherently iterative process, involving multiple rounds of revision to refine ideas, arguments, and language. While AI models have shown promise in text revision, they typically operate in a "one-shot" manner, producing a single set of edits. This approach is limited because it doesn't capture the dynamic, iterative nature of human revision, nor does it effectively incorporate human feedback throughout the process. Applying these one-shot models iteratively can lead to "noisy" edits that burden human writers. The authors address this by developing a system that facilitates a more natural, collaborative, and iterative revision workflow between humans and AI.

Methods

The core of R3 is a human-in-the-loop iterative text revision system. It operates in cycles:

  1. Text Revision System: This component uses a two-step process. First, an edit-prediction classifier determines if a sentence needs revision. If so, an edit-intention classifier assigns a category (FLUENCY, COHERENCE, CLARITY, STYLE) to the intended edit. A fine-tuned PEGASUS language model then generates a revised sentence conditioned on the source sentence and the predicted edit intention.
  2. Human-in-the-loop Interface: Users are presented with suggested edits and their corresponding intentions. They can then choose to accept or reject each edit.
  3. Iterative Process: Accepted edits are incorporated back into the document, and the system generates a new set of revisions for the next iteration. This cycle continues until a stopping criterion is met (e.g., no new edits are generated or a maximum revision depth is reached).

The system is built upon prior work by Du et al. (2022) for edit intention identification and text revision generation, but crucially adds the iterative human-in-the-loop component.

Datasets

  • Training Data: The authors used a newly collected iterative text revision dataset, which is larger than previous datasets, spanning three domains: ArXiv, Wikipedia, and Wikinews. This data was used to train both the edit intention identification models and the text revision generation model.
  • Evaluation Data: For human evaluation, they sampled documents from the test set of their collected dataset. For the HUMAN-HUMAN baseline, they used existing ground-truth references from their dataset.

Key Insights and Ideas

  • Iterative Collaboration is Key: The paper strongly suggests that true text revision assistance requires an iterative, collaborative approach, not just a single-shot AI output.
  • Human Feedback Refines AI: Incorporating user feedback (accepting/rejecting edits) significantly improves the quality of revisions and can lead to better outcomes with fewer iterations.
  • AI Can Match Human Quality Early On: R3's AI-generated edits show comparable acceptance rates to human-written edits at early revision depths, indicating the AI's ability to produce high-quality suggestions.
  • Edit Intentions Matter: While CLARITY edits are frequent, users accept them less often, suggesting room for improvement in AI's ability to generate clear and effective clarity edits. Conversely, STYLE edits, though less common from humans, are surprisingly well-received when generated by the AI.

Key Results

  • Comparable Acceptance Rates: At revision depth 1, R3 generates more edits than human writers and gets more accepted. At revision depth 2, R3 generates fewer edits but achieves a higher acceptance rate than human writers.
  • Improved Revision Quality: The SYSTEM-HUMAN setup (with human-in-the-loop) produced better overall quality scores for final revised documents compared to SYSTEM-ONLY (no human feedback) and even ground-truth human references, with fewer iterations and edits.
  • User Feedback: Users found the system convenient but were neutral on productivity due to a lack of domain knowledge. Retention and satisfaction scores were slightly negative, attributed to UI design issues.

Limitations and Future Work

  • Model Training Data: The models are trained on static human revision data. This might limit their ability to adapt to the evolving nature of revisions in later iterative steps, as seen by the decrease in acceptance rates at revision depth 3.
  • UI/UX Improvements: User feedback highlighted areas for improvement in the user interface, such as alignment issues and presentation of word-level edits.
  • Limited User Role: Users were restricted to accepting or rejecting edits. Future work could explore more advanced interaction interfaces, allowing users to rewrite machine-revised text.
  • Larger-Scale Studies: The authors suggest conducting larger user studies to derive more meaningful statistics on optimal revision depths and edit suggestion strategies, and to investigate user behavior with their own texts versus third-party texts.
  • Developing Better Models: Future work could focus on developing more powerful text revision generation models, particularly to bridge the gap in FLUENCY edits where human revisions are significantly more accepted than system-generated ones.

Summary updated

Abstract

Revision is an essential part of the human writing process. It tends to be strategic, adaptive, and, more importantly, iterative in nature. Despite the success of large language models on text revision tasks, they are limited to non-iterative, one-shot revisions. Examining and evaluating the capability of large language models for making continuous revisions and collaborating with human writers is a critical step towards building effective writing assistants. In this work, we present a human-in-the-loop iterative text revision system, Read, Revise, Repeat (R3), which aims at achieving high quality text revisions with minimal human efforts by reading model-generated revisions and user feedbacks, revising documents, and repeating human-machine interactions. In R3, a text revision model provides text editing suggestions for human writers, who can accept or reject the suggested edits. The accepted edits are then incorporated into the model for the next iteration of document revision. Writers can therefore revise documents iteratively by interacting with the system and simply accepting/rejecting its suggested edits until the text revision model stops making further revisions or reaches a predefined maximum number of revisions. Empirical experiments show that R3 can generate revisions with comparable acceptance rate to human writers at early revision depths, and the human-machine interaction can get higher quality revisions with fewer iterations and edits. The collected human-model interaction dataset and system code are available at \url{https://github.com/vipulraheja/IteraTeR}. Our system demonstration is available at \url{https://youtu.be/lK08tIpEoaE}.
BibTeX
@inproceedings{du2022read,
  author = {Wanyu Du and Zae Myung Kim and Vipul Raheja and Dhruv Kumar and Dongyeop Kang},
  title = {{Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revision}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure