How Learners Detect Revision Occasions in Texts Labeled as Peer-Written or AI-Generated

Anna Radtke, Qian Ruan, Iryna Gurevych, Nikol RummelProceedings 2025 · DOI

Abstract

As Large Language Models (LLMs) become integrated into academic writing, the focus shifts from text generation to the critical process of revision. This study investigated whether learners (N = 152) detect different "revision occasions" when a text is labeled as either peer-written or AI-generated. While the content was identical across conditions, participants descriptively identified fewer surface-level errors (grammar, clarity) in the AI-labeled text. No significant difference was found for semantic revisions. These findings suggest a subtle "AI halo effect" that may lead learners to under-critique the stylistic quality of machine-generated prose.

Introduction

Revision is a cornerstone of the writing process, serving as a "learning tool" that allows students to internalize content and apply linguistic rules (Rijlaarsdam et al., 2004). When learners revise texts they did not write, they gain exposure to new perspectives and arguments. In the era of generative AI, students are increasingly tasked with revising LLM-generated drafts. However, AI-generated texts often contain "hallucinations" or stylistic inconsistencies that require rigorous oversight. This study explores whether the mere label of "AI" acts as a cognitive bias, altering how learners perceive and improve a text.

Methodology

The researchers conducted a controlled, between-subjects experiment with 152 participants. All participants were asked to revise an identical 3.5-page academic text concerning the relationship between sleep and life quality. The experimental manipulation involved the author label: half the participants were told the text was written by a peer, while the other half were told it was AI-generated.

Revisions were categorized using an adjusted version of the Edit Intent Classification framework (Ruan et al., 2024). Revision occasions were divided into two primary categories:

  1. Surface-level: Focused on grammar, clarity, and style/tone.
  2. Semantic: Focused on facts, evidence, claims, and coherence.

The study utilized the ICAP framework (Interactive, Constructive, Active, Passive) to interpret these behaviors. Surface revisions are associated with "Active" learning, whereas semantic revisions—which involve deeper cognitive engagement with the text's meaning—are associated with "Constructive" learning.

Results and Analysis

The analysis revealed that the author label did not have a statistically significant overall effect on the combination of revision types (Wilk’s $\Lambda = .973, p = .131$). However, a notable descriptive trend emerged regarding surface-level critiques.

The Surface-Level Trend

Participants in the "Peer" condition identified an average of 34.83 surface-level revision occasions, compared to 29.32 in the "AI" condition. Although this result was not statistically significant ($F(1,150) = 2.854, p = .093$), the trend suggests that learners may be less critical of the mechanics and clarity of text when they believe it was produced by an AI.

Semantic Consistency

In contrast, detection of semantic issues was nearly identical between the two groups, with both identifying roughly the same number of factual or logical errors. This indicates that while learners might be slightly biased toward AI prose style, they remain equally capable of identifying deep-seated content issues regardless of the purported author.

Discussion

The descriptive findings point toward a potential "AI halo effect," where the perceived technical sophistication of LLMs leads learners to overlook stylistic flaws. From the perspective of the ICAP framework, failing to engage with surface-level errors prevents learners from fully entering the "Active" mode of learning. Furthermore, the lack of increased semantic engagement suggests that the AI label does not currently prompt learners to move into a "Constructive" or "Interactive" mode where they act as co-creators of ideas.

Limitations and Future Work

A key limitation of this study was the low inter-rater reliability for specific sub-categories like "Fact/Evidence" and "Claim," highlighting the difficulty of distinguishing these elements in academic writing. Additionally, the study used a static label. Future research should investigate how learners detect revision occasions during dynamic, iterative interactions with AI agents, which may better facilitate the "Interactive" mode of the ICAP framework.

Conclusion

As AI becomes a standard collaborator in education, it is vital that learners maintain a critical stance. The tendency to under-critique AI-generated style suggests that educational interventions should focus on sharpening students' critical evaluation skills, ensuring they treat AI drafts with the same rigor as any human-authored text.

Summary updated

Abstract

Although the potential of large language models in education has been widely recognized, research on how learners revise AI-generated content remains limited.The aim of this study was to determine whether learners detect different revision occasions when revising texts labeled as written by different authors (peer vs. AI).In a controlled study, learners (N = 152) revised a text that was labeled as either peer-written or AI-generated, while the content of this text was in fact identical in both conditions.We found that learners descriptively detected fewer surface-level revision occasions in a text labeled as AI-generated, although this difference was not statistically significant.No notable difference was found for semantic (i.e., meaning-changing) revision occasions. Revision occasions in peer-written and AI-generated textsA substantial body of research has examined how learners revise their own academic texts and those written by their peers (e.g., Rijlaarsdam et al., 2004).Recently, initial empirical evidence on learners' behaviors when revising AI-generated texts has been presented (e.g., Radtke & Rummel, 2025).However, further research is needed to examine learners' approaches to revising AI-generated content, particularly with regard to whether learners exhibit specific foci and deficits in their revision that should be considered in educational practice and research. Revision of peer-written and AI-generated textsRevision is considered a crucial stage in the writing process.In addition to improving the quality of a text, revision can serve as a 'learning tool' (Rijlaarsdam et al., 2004, p. 199): While revising a text, learners can engage in a comprehensive evaluation and internalization of its content, recall and apply previously learned rules, or even acquire new revision skills through practice.Moreover, when learners revise texts that they have not written in the first place, they can gain insight into new facts and arguments developed by their peers (Fitzgerald, 1987;Rijlaarsdam et al., 2004).Furthermore, empirical evidence suggests that revising peer-written texts can also enhance learners' individual writing and revision skills (e.g., Wichmann & Rummel, 2013).The advent of generative artificial intelligence (AI), particularly large language models (LLMs) such as GPT or BLOOM, in the field of education has major implications for learning practices (e.g., Kasneci et al., 2023).Given the capacity of LLMs to generate texts of a high quality that closely resemble the style of human writers, students worldwide are keen to use this technology, especially in academic writing (e.g., Albayati, 2024).Nevertheless, despite the initial impression of polished and coherent content, a closer examination often reveals that AIgenerated texts require substantial revision.For instance, there is a risk that AI-generated texts contain incorrect information (e.g., hallucinations; Marcus & Davis, 2020) or biased content (OpenAI, 2024).Additionally, it is not implausible that AI-generated texts may contain grammatical mistakes, particularly in non-English texts (Advertext, 2023).However, given the relative novelty of the phenomenon of AI-assisted writing, there is currently a lack of empirical evidence examining how learners engage in the revision of AI-generated academic texts.The processes of revising peer-written and AI-generated texts appear similar in that in both cases, learners are engaged in revising a text that they have not written in the first place.A further rationale for comparing these processes is the growing body of research that draws parallels between collaborative writing and AI-assisted writing in general, with revision being an essential component of both of these learning situations (e.g., Cress & Kimmerle, 2023).In this study, we provided participants with the same text to be revised, varying only the author label (peer vs. AI) between the experimental conditions.This allowed us to determine whether learners exhibit different patterns of detecting revision occasions when revising purportedly peer-written or AI-generated texts. Classification of revision occasionsWhen revising an academic text, learners need to employ two main revision skills: (1) detecting areas that require improvement (revision occasions) and (2) applying the most appropriate improvement strategies (Hayes, 2004).In this paper, we deliberately use the term revision occasions in order to emphasize the following: The revision
BibTeX
@inproceedings{radtke2025how,
  author = {Anna Radtke and Qian Ruan and Iryna Gurevych and Nikol Rummel},
  title = {{How Learners Detect Revision Occasions in Texts Labeled as Peer-Written or AI-Generated}},
  booktitle = {Proceedings},
  year = {2025},
  doi = {10.22318/icls2025.473453},
}
Figure