Span Identification of Epistemic Stance-Taking in Academic Written English

Masaki Eguchi, Kristopher KyleACL 2023

TL;DR

This paper introduces a novel approach to automatically identify "stance-taking" expressions in academic writing, moving beyond simple sentiment analysis to capture how authors position themselves and engage with different viewpoints. They developed a new annotated dataset and a machine learning model that achieves performance comparable to human annotators, paving the way for more nuanced automated writing evaluation systems.

What Problem Does This Paper Address?

Automated Writing Evaluation (AWE) systems are increasingly used to provide feedback on student writing. However, current systems often focus on surface-level features like grammar and vocabulary, neglecting deeper rhetorical and pragmatic aspects of writing. One such crucial aspect is "stance-taking" or "evaluative language," which refers to how writers express their opinions, beliefs, and attitudes towards a topic, and how they engage with other voices or potential counterarguments. While discourse analysis has extensively studied these phenomena, there's a lack of computational tools to automatically identify and categorize them at scale. This gap limits the ability of AWE systems to provide comprehensive feedback on the quality of argumentation and persuasive writing.

Methods

The authors tackle this problem by framing it as a span identification task. This means the goal is to identify specific segments (spans) of text that express stance and then categorize them into predefined rhetorical stance categories. They draw upon the Appraisal theory's engagement system from discourse analysis to define these categories.

Their approach involves:

  • Annotation Scheme Development: Adapting the engagement system from Appraisal theory to create a set of eight core stance categories (e.g., PROCLAIM, ATTRIBUTE, COUNTER, MONOGLOSS).
  • Dataset Creation: Manually annotating a corpus of academic English texts with these stance categories.
  • Machine Learning Model Development: Experimenting with three neural network architectures for span identification:
  • Baseline spaCy Span Categorizer: A standard implementation using a Transformer embedder.
  • RoBERTa + Bi-LSTM: Adding a Bidirectional Long Short-Term Memory (Bi-LSTM) layer on top of RoBERTa embeddings to capture sequential information.
  • Dual-RoBERTa: Using two sets of Transformer embeddings in parallel, aiming to capture complementary information.
  • Domain Adaptation: Fine-tuning the RoBERTa models on their specific academic English corpus to improve performance.

Datasets

The primary dataset used in this study is the Engagement Discourse Treebank (EDT), which comprises 4,688 sentences (126,411 tokens) manually annotated for eight rhetorical stance categories. This dataset was sampled from various corpora of academic English, including university written assignments and timed essays by both first and second-language writers. The authors note that this dataset is accessible for research.

Key Insights and Ideas

  • Beyond Sentiment: Stance-taking is more complex than simple positive/negative sentiment. It involves nuanced ways of engaging with other voices, acknowledging or rejecting alternative viewpoints, and asserting one's own position.
  • Discourse Semantics Matter: Identifying stance requires understanding the semantic function of language within its context, not just relying on lexical or syntactic patterns. For example, the word "suggest" can be used to attribute an idea or to hedge a writer's own opinion.
  • Bi-LSTM Enhances Stability: The addition of a Bi-LSTM layer to the RoBERTa model significantly improved its performance and stability, suggesting that capturing sequential information is crucial for this task.
  • Human Annotation is Challenging: Even with extensive training, achieving high inter-annotator agreement for stance categories can be difficult, particularly for complex categories like ATTRIBUTION and PROCLAIM. This highlights the need for robust computational models.

Key Results

The best-performing model, RoBERTa + LSTM, achieved a macro-averaged F1 score of .7208 for span identification of stance-taking expressions. This slightly outperformed the intercoder reliability estimates (F1 = .6629) before adjudication, indicating that the model can identify stance expressions as well as, or even better than, human annotators in some cases. The RoBERTa + LSTM and Dual-RoBERTa models generally outperformed the baseline spaCy model.

Limitations and Future Work

The authors acknowledge that their approach used a minimal context window (three sentences) for annotation due to practical constraints. They suggest that future research should explore longer units of analysis (e.g., paragraphs, whole documents) to enhance annotation quality and capture more comprehensive discourse semantics. They also plan to update the annotated corpus and expand the scope of annotation. The developed annotation guideline is offered as a resource to guide future methodological improvements in discourse annotation.

Summary updated

BibTeX
@inproceedings{eguchi2023span,
  author = {Masaki Eguchi and Kristopher Kyle},
  title = {{Span Identification of Epistemic Stance-Taking in Academic Written English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure