Abstract
As Large Language Models (LLMs) become integrated into academic writing, the focus shifts from text generation to the critical process of revision. This study investigated whether learners (N = 152) detect different "revision occasions" when a text is labeled as either peer-written or AI-generated. While the content was identical across conditions, participants descriptively identified fewer surface-level errors (grammar, clarity) in the AI-labeled text. No significant difference was found for semantic revisions. These findings suggest a subtle "AI halo effect" that may lead learners to under-critique the stylistic quality of machine-generated prose.
Introduction
Revision is a cornerstone of the writing process, serving as a "learning tool" that allows students to internalize content and apply linguistic rules (Rijlaarsdam et al., 2004). When learners revise texts they did not write, they gain exposure to new perspectives and arguments. In the era of generative AI, students are increasingly tasked with revising LLM-generated drafts. However, AI-generated texts often contain "hallucinations" or stylistic inconsistencies that require rigorous oversight. This study explores whether the mere label of "AI" acts as a cognitive bias, altering how learners perceive and improve a text.
Methodology
The researchers conducted a controlled, between-subjects experiment with 152 participants. All participants were asked to revise an identical 3.5-page academic text concerning the relationship between sleep and life quality. The experimental manipulation involved the author label: half the participants were told the text was written by a peer, while the other half were told it was AI-generated.
Revisions were categorized using an adjusted version of the Edit Intent Classification framework (Ruan et al., 2024). Revision occasions were divided into two primary categories:
- Surface-level: Focused on grammar, clarity, and style/tone.
- Semantic: Focused on facts, evidence, claims, and coherence.
The study utilized the ICAP framework (Interactive, Constructive, Active, Passive) to interpret these behaviors. Surface revisions are associated with "Active" learning, whereas semantic revisions—which involve deeper cognitive engagement with the text's meaning—are associated with "Constructive" learning.
Results and Analysis
The analysis revealed that the author label did not have a statistically significant overall effect on the combination of revision types (Wilk’s $\Lambda = .973, p = .131$). However, a notable descriptive trend emerged regarding surface-level critiques.
The Surface-Level Trend
Participants in the "Peer" condition identified an average of 34.83 surface-level revision occasions, compared to 29.32 in the "AI" condition. Although this result was not statistically significant ($F(1,150) = 2.854, p = .093$), the trend suggests that learners may be less critical of the mechanics and clarity of text when they believe it was produced by an AI.
Semantic Consistency
In contrast, detection of semantic issues was nearly identical between the two groups, with both identifying roughly the same number of factual or logical errors. This indicates that while learners might be slightly biased toward AI prose style, they remain equally capable of identifying deep-seated content issues regardless of the purported author.
Discussion
The descriptive findings point toward a potential "AI halo effect," where the perceived technical sophistication of LLMs leads learners to overlook stylistic flaws. From the perspective of the ICAP framework, failing to engage with surface-level errors prevents learners from fully entering the "Active" mode of learning. Furthermore, the lack of increased semantic engagement suggests that the AI label does not currently prompt learners to move into a "Constructive" or "Interactive" mode where they act as co-creators of ideas.
Limitations and Future Work
A key limitation of this study was the low inter-rater reliability for specific sub-categories like "Fact/Evidence" and "Claim," highlighting the difficulty of distinguishing these elements in academic writing. Additionally, the study used a static label. Future research should investigate how learners detect revision occasions during dynamic, iterative interactions with AI agents, which may better facilitate the "Interactive" mode of the ICAP framework.
Conclusion
As AI becomes a standard collaborator in education, it is vital that learners maintain a critical stance. The tendency to under-critique AI-generated style suggests that educational interventions should focus on sharpening students' critical evaluation skills, ensuring they treat AI drafts with the same rigor as any human-authored text.
Summary updated