TL;DR
This paper introduces DiReC, a novel two-stage framework for classifying tutor identity in educational dialogues. By disentangling content and style representations, DiReC achieves state-of-the-art performance on the BEA 2025 Shared Task, demonstrating the power of this approach for understanding nuanced AI and human tutor behavior.
What Problem Does This Paper Address?
As conversational agents, particularly Large Language Models (LLMs), become increasingly integrated into educational settings, understanding their behavior and distinguishing them from human tutors is crucial. The BEA 2025 Shared Task 5 specifically addresses this by challenging participants to identify which of nine different tutors (a mix of human experts and LLMs) generated a given response. This task is important for evaluating the effectiveness and authenticity of AI tutors, ensuring they provide valuable learning support without being indistinguishable from human educators.
Methods
The core innovation of this paper is the DiReC (Disentangled Contrastive Representation) framework. It employs a two-stage training approach:
- Stage 1: Content Representation Learning: The model first learns to extract semantic and structural content features using a standard cross-entropy loss. During this stage, the "style" projection head is frozen, focusing solely on building a robust content understanding.
- Stage 2: Joint Contrastive and Disentanglement Learning: In the second stage, the model unfreezes the style projection head and introduces two key components:
- Supervised Contrastive Loss (LSupCon): Applied to style embeddings, this encourages responses from the same tutor to have similar style representations, regardless of content.
- Disentanglement Loss (Ldis): This loss penalizes high similarity between the content and style embeddings, forcing them to capture distinct aspects of the response.
This disentanglement aims to create representations that are both discriminative for classification and interpretable, allowing researchers to understand what makes a tutor's style unique. The final classification is performed using a linear classifier, and a CatBoost classifier is also explored for improved performance.
Datasets
The experiments were conducted using data from the MRBench dataset, which contains responses from two human tutors (novice and expert) and seven different LLMs. This dataset was used for the BEA 2025 Shared Task 5.
Key Insights and Ideas
- Disentangling Content and Style is Key: The paper strongly suggests that separating semantic content from stylistic nuances is crucial for accurately identifying tutor identities, especially when dealing with a diverse set of tutors including both humans and various LLMs.
- Staged Training Improves Performance: A two-stage training approach, where content is learned first and style refinement follows, significantly outperforms single-stage training. This suggests that allowing the content pathway to converge before introducing style-specific objectives is beneficial.
- Content Dominates Classification Signal: While style is important for differentiation, the content embedding carries the primary classification signal. Naively prolonging content-only training can lead to overfitting and harm generalization.
- Hybrid Classifiers Can Boost Performance: Replacing the simple linear classifier with a more powerful model like CatBoost, trained on the concatenated content and style embeddings, further improves classification accuracy.
Key Results
The DiReC framework achieved strong results on the BEA 2025 Shared Task:
- The two-stage DiReC model achieved a validation macro-F1 score of 0.9042.
- When combined with a CatBoost classifier and refined using a Hungarian algorithm for post-processing, the system achieved a final macro-F1 score of 0.9172.
- This performance placed the "Two Outliers" team in third place overall in the shared task.
- Ablation studies showed that the content projection head alone achieved a macro-F1 of 0.8845, while the style projection head alone achieved 0.8692, highlighting the importance of content.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Hyperparameter Tuning: Due to time and computational constraints, extensive hyperparameter tuning was not performed. Parameters like contrastive temperature, loss weights, learning rate, and batch size were chosen heuristically.
- Potential for Further Improvement: The authors suggest that systematic tuning of these parameters could lead to further performance gains. Future work could also explore more advanced disentanglement techniques and investigate the interpretability of the learned representations more deeply.
Summary updated