Beyond Aggregate WER: In-domain Adaptation, Learner Speech, Code-Switching, and Speaker Attribution in Online Spanish Learning Classrooms

Olga Kellert, Samhitha Harish, Johnatan E Bonilla, Nemika Tyagi, Phanos Kareen, Nelvin Licona-GuevaraarXiv 2026 · arXiv

TL;DR

This paper demonstrates that aggregate Word Error Rate (WER) is a deceptive metric for educational ASR, as it masks significant performance gaps in learner speech and code-switching. By using Low-Rank Adaptation (LoRA) on a small but highly targeted in-domain classroom corpus, the authors show that open-weight models like MOSS can become competitive with top-tier commercial services, proving that domain-specific interactional context is more valuable than sheer data volume.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) is the backbone of many educational technologies, from accessibility tools to automated proficiency assessment. However, standard ASR systems often struggle in authentic second-language (L2) classrooms. These environments are "noisy" not just acoustically, but linguistically: learners have developing pronunciation and non-target grammar, and both students and teachers frequently code-switch between languages.

The authors argue that a single, aggregate WER score hides these failures. A system might perform excellently on a native-speaking instructor but fail miserably on a student, or it might transcribe words accurately but attribute them to the wrong person (speaker diarization). This paper seeks to provide a "condition-aware" evaluation that looks specifically at speaker roles, language contexts, and speaker attribution.

Methods

The researchers compared six different ASR configurations:

  1. Zoom automated captions: The baseline platform-integrated system.
  2. Whisper large-v3: A popular open-weight model from OpenAI.
  3. Whisper + LoRA: Whisper adapted using Low-Rank Adaptation.
  4. MOSS-Transcribe-Diarize: An open-weight model designed to handle transcription and speaker attribution in a single pass.
  5. MOSS + LoRA: MOSS adapted using the same in-domain data.
  6. Muse Voice Transcribe: A high-performing commercial API from Meta.

The study pipeline is summarized in Figure 1: Study pipeline showing data collection, adaptation, testing, and evaluation phases.

Figure 1: Study pipeline. Whisper and MOSS are adapted with LoRA on 38 human-corrected Zoom sessions, with the recipe fixed by five-fold cross-validation. Six systems are scored on seven held-out sessions against human-corrected references.

A key technical contribution is the use of LoRA for adaptation. LoRA allows for fine-tuning a small fraction of the model's parameters, which helps prevent "catastrophic forgetting"—ensuring the model doesn't lose its ability to recognize English while learning the nuances of Spanish classroom interaction.

Datasets

  • In-domain Corpus: 38 sessions (15.6 hours) of online L2 Spanish interaction between L1 English learners and native-speaking instructors/tutors.
  • Evaluation Set: 7 held-out sessions (3.51 hours) representing various course levels (beginner to advanced).
  • Reference Transcripts: Human-corrected transcripts where transcribers preserved learner errors and non-target forms rather than normalizing them, ensuring the ASR was tested on its ability to capture actual learner production.
  • External Data (for comparison): The authors also tested adding larger out-of-domain bilingual corpora (Bangor Miami) and learner corpora (SPLLOC), but found they did not improve performance over the smaller, matched classroom data.

Key Insights and Ideas

  • Domain Match > Data Volume: One of the most striking findings was that adding thousands of hours of out-of-domain data actually hurt performance. The specific "register" of a classroom—the way instructors and learners interact—is a unique signal that general bilingual corpora don't capture.
  • The "Learner Gap" is Persistent: Across all systems, learner speech remained significantly harder to transcribe than instructor speech. Even the best systems saw a ~10 percentage point jump in WER when switching from teacher to student.
  • Transcription vs. Attribution: A system's ranking in transcription accuracy (WER) doesn't necessarily match its ranking in speaker attribution (DER). For example, while Muse had the lowest WER, the MOSS adapter was superior at correctly identifying who was speaking.
  • Decoding Regimes Matter: For Whisper, the gains from adaptation were only realized when decoding in short 30-second windows (matching its training). When forced into long-form sequential decoding, the adaptation benefits vanished, whereas MOSS (which decodes complete recordings) retained its gains.

Key Results

The commercial Muse system and the MOSS adapter emerged as the leading pair. As shown in Table 2, they achieved the lowest overall WER and CER:

Table 3: WER by speaker role, showing the learner-instructor gap.

However, the disaggregated results in Table 3 highlight the performance disparity between roles. While the MOSS adapter significantly closed the gap compared to the base model, a substantial "Learner Gap" remains for all systems:

Table 3: WER by speaker role, showing the learner-instructor gap.

In terms of speaker attribution, the MOSS adapter achieved the lowest Diarization Error Rate (DER) at 10.66%, outperforming even the commercial Muse system (19.12%) in this specific metric (Table 5):

Table 5: Pooled system-provided DER for long-form ASR systems.

Limitations and Future Work

  • Scale: The evaluation was limited to seven sessions from a single institutional context.
  • Language Pair: The study focused exclusively on Spanish-English interaction.
  • Segmentation Bias: Different systems used different segmentation methods (e.g., Whisper used reference-defined turns while others processed raw audio), which complicates direct architectural comparisons.
  • Future Directions: The authors suggest that future work should explore how these ASR errors impact downstream tasks like automated feedback or discourse analysis, and whether further gains can be made by explicitly modeling the interactional roles of participants.

Summary updated

Abstract

Automatic speech recognition (ASR) increasingly supplies transcripts for educational NLP, but aggregate word error rate (WER) can conceal errors concentrated in learner speech, multilingual turns, and speaker attribution. We compare platform-integrated, open-weight, adapted, and commercial ASR configurations on held-out sessions of online L2 Spanish interaction between L1 English learners and their instructors and tutors. Because English is the learner's shared first language, the interactions contain Spanish-only, English-only, and code-switched turns. Evaluation combines overall WER and character error rate (CER), WER disaggregated by speaker role and language context, and diarization error rate (DER). We also adapt Whisper large-v3 and MOSS-Transcribe-Diarize with LoRA using human-corrected in-domain classroom data. MOSS adaptation produced consistent improvements across speaker roles and language contexts, whereas Whisper showed smaller gains. Domain match proved more important than data volume: Larger out-of-domain bilingual and learner corpora did not outperform the smaller classroom corpus, where learner speech and code-switching are embedded in instructor--learner interaction. The open-weight MOSS adapter was competitive with commercial Muse Voice Transcribe on WER and CER and achieved lower system-provided DER on the recordings eligible for each system. Learner, English-only, and code-switched speech remained more difficult than instructor and Spanish-only speech. The findings demonstrate the value of condition-aware evaluation and interactionally matched data for bilingual educational ASR.
BibTeX
@misc{kellert2026beyond,
  author = {Olga Kellert and Samhitha Harish and Johnatan E Bonilla and Nemika Tyagi and Phanos Kareen and Nelvin Licona-Guevara},
  title = {{Beyond Aggregate WER: In-domain Adaptation, Learner Speech, Code-Switching, and Speaker Attribution in Online Spanish Learning Classrooms}},
  year = {2026},
  eprint = {2610.38016},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38016},
}
Figure