TL;DR
This paper introduces a novel annotation scheme and a freely available corpus for detecting cognitive and emotional empathy in student-written peer reviews. They then leverage this to build and evaluate an adaptive writing support system that provides students with personalized feedback to improve their empathy skills, demonstrating promising results in a peer learning setting.
What Problem Does This Paper Address?
Empathy is a crucial skill for effective communication and collaboration, yet research indicates a decline in empathy among college students. While traditional methods for teaching empathy exist, they often struggle to scale in large educational settings or online courses. Existing computational approaches to empathy detection in text are limited, particularly in educational domains, and often lack rigorous annotation schemes and publicly available datasets. This paper aims to bridge this gap by developing a method to automatically assess and support the development of empathy in student writing.
Methods
The authors developed a multi-stage annotation scheme to capture empathy in peer reviews. First, each review component (strengths, weaknesses, suggestions for improvement) was identified. Second, annotators assigned scores for both cognitive empathy (understanding the other's perspective) and emotional empathy (reacting to the other's feelings) on a 1-to-5 scale, guided by detailed definitions derived from psychological theory.
To build predictive models, they employed a Bidirectional Encoder Representations from Transformers (BERT) architecture, a state-of-the-art deep learning model for natural language processing. This model was trained on their annotated corpus to classify text paragraphs by review component and empathy level. The trained models were then integrated into an adaptive writing support system.
Datasets
The paper introduces a new dataset: 500 student-generated peer reviews in German, collected from a business innovation master's program. These reviews were manually annotated for three types of components (strengths, weaknesses, suggestions for improvement) and for cognitive and emotional empathy levels (1-5 scale). This corpus is made freely available.
Key Insights and Ideas
- Empathy is Multidimensional: The research explicitly models empathy as having both cognitive (understanding) and emotional (feeling) components, aligning with established psychological theories.
- Annotation is Feasible: Despite the subjective nature of empathy, the study demonstrates substantial to moderate inter-annotator agreement, suggesting that empathy can be reliably annotated in student writing with clear guidelines.
- Context Matters: The annotation scheme distinguishes between different review components (strengths, weaknesses, suggestions), recognizing that empathy can manifest differently within each.
- Adaptive Feedback is Promising: An adaptive writing support system that provides personalized, automated feedback on empathy can lead to perceived skill improvement and increased intention to use the tool.
Key Results
- Annotation Reliability: The annotation scheme achieved an inter-annotator agreement of $\alpha=0.79$ for components and multi-$\pi=0.41$ for empathy scores, indicating reliable annotation.
- Model Performance: The BERT model achieved an f1-score of 74.96% for emotional empathy detection and 69.98% for cognitive empathy detection at the paragraph level. For component classification, it achieved an f1-score of 94.83% for strengths, 64.28% for weaknesses, and 59.79% for suggestions.
- User Study: In a controlled evaluation with 58 students, the adaptive writing support tool led to promising results: perceived empathy skill learning (mean 5.03/7), perceived feedback accuracy (mean 4.93/7), and intention to use (mean 5.14/7). Participants positively noted the clear distinction between cognitive and emotional empathy feedback.
Limitations and Future Work
- Domain Specificity: The current corpus and models are specific to business model peer reviews written in German. Future work could explore other domains, languages, and text types.
- Model Granularity: While the system provides feedback on cognitive and emotional empathy, some users requested more granular feedback and detailed examples for improvement.
- Ethical Considerations: The authors acknowledge the importance of ethical considerations in empathy detection models for user-based research.
- Future Directions: The authors hope their work will encourage further research on student-generated empathetic texts and the development of NLP-based writing support systems to enhance empathy skills, making quality education more accessible regardless of location or instructor availability. They also suggest leveraging the corpus for conversational interaction to support learning.
Summary updated