TL;DR
This paper tackles the challenge of automatically identifying the purpose behind revisions in argumentative student writing, moving beyond simply detecting that a revision occurred. They introduce novel contextual features and a sequence labeling approach to capture dependencies between revisions, significantly improving prediction accuracy.
What Problem Does This Paper Address?
Writing assistance tools are increasingly sophisticated, but providing truly helpful feedback requires understanding why an author made a change. While previous work has focused on detecting revisions and classifying their basic type (e.g., addition, deletion), this paper addresses the more nuanced problem of determining the argumentative purpose of these revisions. For instance, was a change made to strengthen a claim, provide better evidence, or clarify reasoning? Existing methods often treat revisions in isolation, ignoring their surrounding context and potential dependencies, which limits their effectiveness in providing targeted suggestions. This is particularly important in argumentative writing, where revisions can subtly alter the flow and persuasiveness of an argument.
Methods
The authors propose two main strategies to improve the prediction of argumentative revision purposes:
- Contextual Features: They extend prior work by incorporating features that capture the revision's local context. This includes:
- Extended Baseline Features: Instead of just looking at the revised sentence pair, they also consider features from the sentences immediately before and after the revision.
- Cohesion and Coherence Features: They hypothesize that revisions impact the text's flow and connectedness. They extract features measuring the lexical and semantic similarity between the revised sentence and its neighbors (cohesion/coherence scores), as well as features quantifying the change in these scores before and after the revision.
- Sequence Labeling: To address the dependencies between revisions, they transform the problem into a sequence labeling task. They sort sentences by their order in the essay and treat aligned sentence pairs (representing revisions) as elements in a sequence. This allows them to use Conditional Random Fields (CRFs) to model the relationships between consecutive revisions, which is crucial because a revision's purpose might be influenced by previous or subsequent changes.
Datasets
The experiments were conducted on two corpora of student essays:
- Corpus A: 47 paper draft pairs about placing contemporaries in Dante's Inferno.
- Corpus B: 63 paper draft pairs explaining rhetorical strategies in texts.
These corpora were double-coded, and gold-standard labels were established through annotator agreement. The revision purpose schema is adapted from prior work and includes categories like Claims/Ideas, Warrant/Reasoning/Backing, Evidence, General Content, and Surface.
Key Insights and Ideas
- Context is King for Revision Purpose: Simply looking at a revision in isolation is insufficient. Capturing the surrounding text and how a revision affects text flow (cohesion/coherence) significantly boosts performance.
- Revisions are Not Independent Events: The sequential nature of writing and revision means that changes are often related. Treating revisions as a sequence, rather than independent events, unlocks better predictive power.
- CRFs Outperform Traditional Classifiers: The sequence labeling approach using CRFs consistently outperformed traditional classifiers (like SVMs) when incorporating contextual features, highlighting the importance of modeling revision dependencies.
- Argumentative Structure Matters: The paper's findings suggest that understanding the argumentative structure of text is key to interpreting revision purposes, especially when revisions relate to core claims.
Key Results
- The proposed contextual features (both "Ext" and "Coh") showed significant improvements over baseline features, particularly in Corpus B.
- The CRF approach, leveraging sequence modeling and contextual features, consistently yielded the best results across both corpora, outperforming SVMs.
- The "Coh" features (measuring cohesion and coherence changes) were particularly effective, showing significant improvements.
- The paper notes that while the "Ext" features offered a slight improvement, combining "Ext" and "Coh" features did not further improve SVM performance, suggesting that "Coh" features might capture some of the information from "Ext."
Limitations and Future Work
- Limited Context Window: The current "Coh" features consider only the immediately adjacent sentences. The authors plan to investigate whether including more sentences in the context window can further improve performance.
- Corpus Specificity: The study focused on student argumentative writing. Future work could explore whether these revision dependencies exist in other types of corpora, such as Wikipedia revisions.
- Difficulty with General/Warrant Distinction: The error analysis revealed that differentiating between "General Content" and "Warrant" revisions remains a significant challenge for both SVM and CRF approaches.
- Data Availability: The corpora used in this study cannot be published due to IRB restrictions, but the authors are initiating a user study project to develop a web-based revision assistant and plan to publish data from that project.
Summary updated