TL;DR
This paper introduces a novel taxonomy and a new dataset, MANUSCRIPT, to better understand the end-to-end writing process in scholarly manuscripts. By categorizing writer intentions and actions, they aim to enable more sophisticated writing assistants that can offer better feedback throughout the entire writing lifecycle.
What Problem Does This Paper Address?
Academic writing is a complex, iterative process involving planning, drafting, and revising. While large language models (LLMs) have shown promise in text generation and revision, they often struggle with providing high-level, structural, and creative feedback crucial for academic work. Existing tools and datasets tend to focus on specific stages of writing (e.g., revision history, text generation) rather than capturing the holistic, non-linear trajectory of a scholarly manuscript's creation. This gap limits the ability of AI writing assistants to offer truly comprehensive support.
Methods
The authors propose a hierarchical taxonomy to categorize scholarly writing behaviors. This taxonomy has three levels:
- Intention: The overarching goal of the writer (Planning, Implementation, Revision).
- Writer Actions: Specific activities undertaken to achieve the intention (e.g., idea generation, organization, lexical chaining, syntactic/lexical/structural revisions).
- Information Types: The linguistic units being manipulated (e.g., sentences, paragraphs, sections, math, figures).
They developed a Chrome extension to capture user keystrokes and infer writing actions in real-time from Overleaf editing history. This data was then annotated using a simplified version of their taxonomy.
Datasets
The paper introduces MANUSCRIPT, a new dataset comprising four scholarly writing trajectories. These trajectories were collected from a pilot study of four participants over 30-minute sessions, resulting in 46 discontinuous edits and 3290 recorded actions. The dataset is annotated with a simplified two-level schema (intention and writer actions).
Key Insights and Ideas
- Writing is Non-Linear: The study reinforces the idea that writing is not a strictly sequential process. Writers frequently move back and forth between planning, implementing, and revising, often embedding sub-processes within others.
- The Need for End-to-End Understanding: To provide effective assistance, writing tools need to understand the writer's current intention and action within the broader context of the entire writing process, not just isolated edits.
- LLMs' Planning Deficit: The paper highlights that LLMs still struggle with higher-level planning, a critical component of academic writing, which limits their ability to aid writers effectively in this phase.
- Annotation Challenges: Even with clear guidelines, annotating writing actions can be ambiguous, especially when distinguishing between early-stage planning and implementation, or when a writer's process deviates significantly from typical patterns (e.g., writing linearly without explicit document planning).
Key Results
- The MANUSCRIPT dataset contains 46 discontinuous edits and 3290 recorded actions.
- Inter-annotator agreement (mean F1) across three annotators was 65.26%, with some challenges noted in distinguishing between "Planning - Generation" and "Implementation - Lexical Chaining" for certain writing styles.
- Analysis of participant trajectories revealed varying degrees of engagement across Planning, Implementation, and Revision phases, with some participants spending the majority of their time in Planning. Participant 2 was identified as an outlier, potentially implementing an internal plan rather than explicit document planning.
Limitations and Future Work
- Limited Study Scope: The current study design is limited by its short duration (30 minutes) and a constrained prompt, which may not fully capture the nuances of real-world scholarly writing.
- Simplified Schema: The initial annotation schema is simplified. Future work will involve applying a more complex, hierarchical taxonomy (as illustrated in Figure 2) to capture finer-grained details.
- Scaling Data Collection: The authors plan to collect more data over longer periods and from more participants to validate their taxonomy and gain deeper insights.
- Collaborative Writing: Future research will explore how writing trajectories differ in collaborative writing environments.
- Cross-Disciplinary Nuances: The current taxonomy is biased towards computer science writing conventions. Further study is needed to adapt it for other academic disciplines, particularly for the "Implementation" phase.
- Audio Component: Incorporating an audio component where participants narrate their process is suggested to improve annotation confidence and understanding of intentions.
Summary updated