SYSUpporter Team at BEA 2025 Shared Task: Class Compensation and Assignment Optimization for LLM-generated Tutor Identification

Longfeng Chen, Zeyu Huang, Zheng Xiao, Zeng YaWen, Jin Xu2025 · DOI

Abstract

In this paper, we propose a novel framework for the tutor identification track of the BEA 2025 shared task (Track 5). Our framework integrates data-algorithm co-design, dynamic class compensation, and structured prediction optimization. Specifically, our approach employs noise augmentation, a fine-tuned DeBERTa-v3-small model with inverse-frequency weighted loss, and Hungarian algorithm-based label assignment to address key challenges, such as severe class imbalance and variable-length dialogue complexity. Our method achieved a 0.969 Macro-F1 score on the official test set, securing second place in this competition. Ablation studies revealed significant improvements: a 9.4% relative gain in robustness from data augmentation, a 5.3% boost in minority-class recall thanks to the weighted loss, and a 2.1% increase in Macro-F1 score through Hungarian optimization. This work advances the field of educational AI by providing a solution for tutor identification, with implications for quality control in LLM-assisted learning environments.

Introduction

As Large Language Models (LLMs) become staples in digital tutoring, the line between human-crafted pedagogical support and AI-generated responses has blurred. While general "AI vs. Human" detectors exist, they often fail in the nuanced world of education where LLMs are specifically tuned to mimic expert human behavior.

The BEA 2025 Shared Task (Track 5) addresses this by requiring the identification of the specific source of a tutor's response across nine categories: two human (Expert and Novice) and seven distinct LLMs (Sonnet, Llama 3.1-8B, Llama 3.1-405B, GPT-4, Mistral, Gemini, and Phi-3). This task presents three primary challenges:

  1. Class Imbalance: Certain categories, particularly "Novice" human tutors, are significantly underrepresented in the training data.
  2. Subtle Linguistic Patterns: High-end LLMs mimic expert human tutors so closely that traditional classifiers struggle to differentiate them.
  3. Dialogue Complexity: Responses vary significantly in length and context, complicating consistent feature extraction.

Methodology

The SYSUpporter framework, "Class Compensation and Assignment Optimization," is built on three pillars designed to address these challenges.

Noise Augmentation and Model Selection

To prevent overfitting on the relatively small dataset (2,476 training samples), we implemented noise augmentation. This involved doubling the training data by injecting random perturbations—inserting, deleting, replacing, or swapping words.

After evaluating several architectures, including XGBoost and Random Forest, DeBERTa-v3-small was selected as the backbone. Interestingly, the "small" variant outperformed the "base" model in validation tests, suggesting that for specific linguistic identification tasks, a more compact, well-regularized model is more effective.

Class Compensation

To handle the severe class imbalance, we utilized an inverse-frequency weighted cross-entropy loss. This "class compensation" strategy forces the model to penalize errors on minority classes (like "Novice" humans) more heavily, improving recall for rare categories.

Assignment Optimization via Hungarian Algorithm

A key insight was that the test set dialogues follow a structured distribution (e.g., a fixed set of AI and human responses per conversation). Instead of classifying responses in isolation, we treated inference as a matching problem. By applying the Hungarian algorithm to solve a linear assignment problem, the model ensures that predicted labels match the known distribution of tutors in a conversation, maximizing overall confidence across the dialogue group.

Overview of the SYSUpporter framework, illustrating the two-stage process of model training with noise augmentation and inference using the Hungarian algorithm.

Results and Discussion

The proposed framework achieved a Macro-F1 score of 0.969 and an accuracy of 0.966 on the official test set, earning the team second place in the competition.

Ablation studies highlight the impact of each component. Noise augmentation provided a massive 9.4% relative gain in performance. The weighted loss strategy boosted minority-class recall by 5.3%, while the Hungarian optimization added a final 2.1% boost to the Macro-F1 score. Furthermore, an ensemble strategy using $k=5$ pre-trained models further stabilized predictions compared to a single model.

Ablation study results showing the incremental performance gains from noise augmentation, weighted loss, and Hungarian optimization.

Conclusion

The SYSUpporter framework demonstrates that combining robust data augmentation with structured prediction can effectively distinguish between human and AI-generated educational content. While the current approach is tailored to the specific distribution of the BEA 2025 task, it provides a strong foundation for quality control in LLM-assisted learning. Future work will focus on generalizing the assignment optimization for variable tutor distributions and testing the model in live, fluid educational environments.

Summary updated

Abstract

In this paper, we propose a novel framework for the tutor identification track of the BEA 2025 shared task (Track 5).Our framework integrates data-algorithm co-design, dynamic class compensation, and structured prediction optimization.Specifically, our approach employs noise augmentation, a fine-tuned DeBERTa-v3small model with inverse-frequency weighted loss, and Hungarian algorithm-based label assignment to address key challenges, such as severe class imbalance and variable-length dialogue complexity.Our method achieved 0.969 Macro-F1 score on the official test set, securing second place in this competition.Ablation studies revealed significant improvements: a 9.4% gain in robustness from data augmentation, a 5.3% boost in minority-class recall thanks to the weighted loss, and a 2.1% increase in Macro-F1 score through Hungarian optimization.This work advances the field of educational AI by providing a solution for tutor identification, with implications for quality control in LLM-assisted learning environments.
BibTeX
@misc{chen2025sysupporter,
  author = {Longfeng Chen and Zeyu Huang and Zheng Xiao and Zeng YaWen and Jin Xu},
  title = {{SYSUpporter Team at BEA 2025 Shared Task: Class Compensation and Assignment Optimization for LLM-generated Tutor Identification}},
  year = {2025},
  doi = {10.18653/v1/2025.bea-1.83},
}
Figure