TL;DR
While post-editing large language model (LLM) drafts helps writers inject their personal style and reduce obvious AI markers, embedding-based metrics reveal that a persistent "LLM stylistic fingerprint" remains. Crucially, a significant gap exists between human perception and algorithmic measurement: writers perceive their post-edited text as fully authentic, completely unaware of the residual AI traces detected by style models.
What Problem Does This Paper Address?
As LLMs become ubiquitous for routine workplace communication, their role in personal writing—such as wedding vows, apologies, or condolence letters—remains highly controversial. In these contexts, authenticity and personal style are paramount, and using unedited AI text often feels disingenuous or "icky."
Prior work shows that writers resist AI assistance during the drafting phase of personal text because it directly shapes stylistic expression. This paper investigates whether a collaborative workflow—where an LLM generates an initial draft based on user-provided details, and the user post-edits it—can successfully "salvage" the text, allowing writers to effectively reintroduce their unique personal style.
Methods
The authors conducted a pre-registered, IRB-approved online user study ($n=81$). The experimental workflow split tasks into two main blocks:
- Control Block: Participants planned details and wrote their text entirely from scratch.
- Treatment Block: Participants provided bulleted content details, an LLM (GPT-4o-mini) generated a draft, and participants spent at least two minutes post-editing it to reflect their personal voice.
To quantify "style" objectively, the authors used LUAR (Learning Universal Authorship Representations) embeddings, selecting it after it outperformed five other style embedding models on a proxy authorship identification task. They measured stylistic similarity using cosine distance between these embeddings. Additionally, they utilized PANGRAM, an AI-generated text detector, to evaluate the likelihood of a text being AI-authored. Finally, participant surveys captured subjective perceptions of style, authenticity, and usability.
Datasets
The study generated a novel dataset consisting of writing samples from 81 English-fluent participants. Each participant completed six writing tasks selected from a pool of eight personal scenarios (e.g., wedding vows, apology letters, thank-you notes, catch-up letters). The final dataset includes the participants' unassisted control texts, the raw LLM-generated drafts, the final post-edited texts, and associated survey responses. The data is publicly available on GitHub.
Key Insights and Ideas
- The Illusion of Authenticity: There is a stark disconnect between how text feels to a human and how it looks to a style model. Writers rated their post-edited text as just as representative of their personal style as their purely human-written control text. However, embedding models showed the post-edited text was still heavily anchored to the LLM's default style.
- Stylistic Homogenization: LLMs act as a stylistic magnet. Even after human editing, the resulting texts are significantly more homogeneous (similar to each other) than purely human-written text. The AI leaves a structural or lexical imprint that humans fail to fully erase.
- Idiosyncrasies as Style: Qualitative analysis revealed that "successful" personalization often relied on non-standard language habits. For instance, a participant who systematically omitted spaces after commas in their control text naturally brought that habit into the post-edited text, which significantly boosted the model-measured stylistic similarity to their own voice.
Key Results
- Shift Toward Self: Post-editing successfully moved the text toward the author's voice. The stylistic similarity to the participant's control text significantly increased ($p = .0002, g = 0.55$), while similarity to the raw LLM text decreased ($g = -0.41$).
- The Persistent AI Anchor: Despite this shift, post-edited text remained significantly closer in style to the original LLM text than to the participant's own unassisted writing ($g = -1.43$).
- Weak Correlation: Human perception of style alignment only weakly correlated with the embedding-based metrics ($r = 0.244, p < .0001$).
- User Preference: Despite the lingering AI traces, 58 out of 81 participants stated they would prefer to use the post-editing workflow for future personal writing tasks, citing reduced mental demand and increased efficiency compared to writing alone.
Limitations and Future Work
A primary limitation is that style embedding models like LUAR are trained on the assumption that a single text has a single author with a consistent style; they may not perfectly capture the nuances of hybrid human-AI text. Furthermore, the study relied on simulated tasks with fake details to preserve privacy, which might alter real-world writing behavior.
The authors suggest that future work should explore how audiences (the recipients of these personal letters) perceive post-edited text, as they might be more sensitive to residual AI markers than the authors themselves. Additionally, they recommend developing interface interventions that explicitly surface residual LLM stylistic features to help users edit more precisely.
Summary updated