EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts

Nicy Scaria, Silvester John Joseph Kennedy, Thomas Latinovich, Deepak N. Subramani

Introduction

Evaluating language proficiency according to the Common European Framework of Reference for Languages (CEFR) is traditionally a labor-intensive process requiring human experts. In the rapidly growing field of e-learning, this creates a massive scalability bottleneck. While Large Language Models (LLMs) show promise in general text tasks, they often struggle with the nuanced, multi-criteria rubrics used in formal assessments, such as "Discourse Management" or "Interactive Communication."

Furthermore, there is a distinct lack of publicly available, high-quality datasets for CEFR-aligned speaking assessments, particularly those tailored to specific regional contexts. The EvalYaks project addresses these gaps by introducing a suite of instruction-tuned 7B-parameter models designed to automate the scoring of CEFR B2 English speaking assessments. By leveraging expert-validated synthetic datasets and Parameter-Efficient Fine-Tuning (PEFT), the researchers developed a specialized system that achieves high grading consistency and rubric alignment.

Methodology

The authors followed a multi-stage development pipeline to create the EvalYaks family of models:

  1. Synthetic Data Generation: Using GPT-4 Turbo, they generated 3,060 simulated conversation records across the four parts of the Cambridge B2 First speaking exam. They utilized Chain-of-Thought (CoT) prompting to ensure the model "reasoned" through the scores.
  2. Expert Alignment: Human experts (English teachers with 4+ years of experience) validated and adjusted the synthetic data to ensure the transcripts perfectly matched the assigned CEFR scores.
  3. Instruction Tuning: The team performed fine-tuning using LoRA (Low-Rank Adaptation) on the Mistral Instruct 7B v0.2 base model.
  4. Specialized Adapters: Instead of one monolithic model, they trained specific LoRA adapters for different tasks: four for the specific sections of the B2 exam, one for vocabulary level detection, and one for sentence-level CEFR alignment.
Comparison of the manual Cambridge CEFR B2 assessment framework and the automated EvalYaks scoring pipeline.

As shown in the figure above, the EvalYaks pipeline (b) automates the traditional human-led framework (a) by processing transcripts through specialized models that apply established assessment criteria to generate final scores.

Datasets

The study utilized three primary data sources to ensure robust training and evaluation:

  • CEFR B2 Speaking Dataset (New): 3,060 expert-validated synthetic data points covering four exam parts: Conversation, Individual Long Turn, Collaborative Task, and Discussion.
  • English Vocabulary Profile (EVP): 3,072 instruction data points for identifying and generating CEFR-level-specific vocabulary.
  • CEFR-SP WikiAuto: A dataset of 17,000 sentences used to create 19,142 instruction points for sentence-level proficiency detection.

Key Results and Insights

EvalYaks demonstrated significant improvements over general-purpose commercial models in both accuracy and consistency.

Performance Metrics

  • High Accuracy: EvalYaks achieved an average acceptable accuracy of 96%, defined as being within one point of the reference score.
  • Low Variance: The Degree of Variation (DOV)—a measure of how far the model's scores stray from the ground truth—was only 0.35, compared to 1.00 for Gemini Pro 1.0 and 1.79 for the base Mistral model.
  • Superiority: EvalYaks performed roughly three times better than the next best commercial model in terms of accurate score prediction.

Key Insights

  • Domain Specificity Trumps Scale: A fine-tuned 7B model outperformed models with potentially hundreds of billions of parameters (like Gemini Pro 1.0 and GPT-3.5) on this specific task. This reinforces the idea that for specialized rubrics, "small" tuned models are more efficient than "large" general ones.
  • The "Spiky Profile" Challenge: Language learners often excel in one area (e.g., vocabulary) while failing in another (e.g., interactive flow). The authors specifically calibrated their training data to include these "rare" score combinations to ensure the model didn't just learn to give uniform scores across all criteria.
  • Contextual Boosting: While off-the-shelf models required the inclusion of performance descriptors in the prompt to improve accuracy, EvalYaks internalized these rubrics so well during tuning that it maintained 96% accuracy even without them.

Limitations and Future Work

While EvalYaks represents a significant step forward, several limitations remain. The current system is text-only and cannot evaluate pronunciation, which is a core component of speaking assessments. Future work aims to develop a multi-modal system (audio + text) to address this.

Additionally, while expert-validated, the training data is still synthetic. Testing on a large corpus of real-world student recordings will be necessary to further validate the model's robustness. Finally, the authors plan to incorporate Direct Preference Optimization (DPO) and reflection agents to provide learners with detailed qualitative feedback alongside numerical scores.

Summary updated

BibTeX
@inproceedings{scaria2025evalyaks,
  author = {Nicy Scaria and Silvester John Joseph Kennedy and Thomas Latinovich and Deepak N. Subramani},
  title = {{EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts}},
  booktitle = {Computers},
  year = {2025},
  doi = {10.1016/j.caeai.2025.100539},
}
Figure