Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

Anand Syamkumar, Nora Tseng, Kaycie Barron, Shanglin Yang, Shamya Karumbaiah, Rheeya Uppal, Junjie Hu

TL;DR

This paper identifies a significant performance gap in how large language models (LLMs) grade student writing that uses "Spanglish" (code-switching between English and Spanish). By fine-tuning open-source models like Llama 3.1 on small, high-quality synthetic bilingual datasets, the researchers eliminated this bias, achieving near-perfect grading accuracy across all three linguistic modes.

What Problem Does This Paper Address?

As LLMs are increasingly integrated into classrooms for automated grading and feedback, there is a growing risk of "linguistic bias." Most models are trained predominantly on monolingual English data. This creates a barrier for bilingual students who practice translanguaging—the fluid use of multiple languages (like Spanglish) to express complex ideas.

The authors point out that if a model can grade an English explanation correctly but fails when the same idea is expressed in Spanglish, the system is fundamentally inequitable. The core challenge is "data scarcity": there aren't many high-quality, labeled datasets of student code-switching available to train these models.

Methods

The researchers employed a multi-stage experimental pipeline:

  1. Synthetic Data Generation: They used Claude 3.5 Sonnet to generate a parallel dataset of Science and Social Science questions and answers in English, which were then translated into Spanish and Spanglish.
  2. Evaluation of Pre-trained Models: They established a baseline by testing the "zero-shot" performance of two open-source models: Llama 3.1 (8B) and Mistral NeMo (12B).
  3. Fine-tuning: They used LoRA (Low-Rank Adaptation) to fine-tune these models. LoRA allows for efficient training by only updating a small fraction of the model's parameters.
  4. Cross-lingual Transfer Analysis: They tested how fine-tuning on one language (e.g., just Spanglish) affected performance in others (e.g., English or Spanish) to see if the models were learning underlying concepts or just linguistic patterns.

Datasets

The study introduced a new synthetic dataset of 1,300 samples (split into 150 for training, 150 for validation, and 1,000 for testing).

  • Content: Middle and high school Science and Social Science concepts.
  • Labels: Binary grades (Acceptable vs. Unacceptable).
  • Validation: To ensure the synthetic data wasn't "hallucinated" or low-quality, human experts (fluent in both languages) reviewed a subset, rating the labels at 99% correctness and the Spanglish "naturalness" at 98.7%.

Key Insights and Ideas

  • The "Hybridization" Penalty: Even though MLLMs are proficient in English and Spanish separately, they struggle significantly when the two are mixed. This suggests that "multilingual" doesn't automatically mean "code-switching capable."
  • Spanglish as a "Super-Language": One of the most surprising findings was that fine-tuning a model only on Spanglish data led to massive improvements in English and Spanish performance as well. The model essentially learned to be more robust across the entire linguistic spectrum.
  • Quality Over Quantity: Significant performance gains were achieved with only 150 training examples. This suggests that for niche educational tasks, massive datasets may be less important than highly representative, nuanced examples.
  • Mixed-Language Training is Best: The most stable performance across all student groups came from "mixed" datasets (e.g., English + Spanglish). This prevents the model from "forgetting" one language while learning another.

Key Results

  • Baseline Bias: In zero-shot tests, Llama 3.1 achieved an AUC of 0.90 for English but only 0.74 for Spanglish. Mistral NeMo showed a similar drop (0.84 to 0.69).
  • Fine-tuning Success: After fine-tuning on Spanglish data, Llama 3.1’s performance on Spanglish jumped from 0.74 to 0.955 AUC.
  • Cross-lingual Transfer: A model fine-tuned on Spanglish achieved an AUC of 0.970 on English tasks, actually outperforming the model fine-tuned on English alone (0.885). This highlights the "robustness" gained by training on hybridized language.

Limitations and Future Work

  • Synthetic Reliance: The primary limitation is the use of synthetic data. While human-validated, synthetic Spanglish may still differ from the "authentic" Spanglish used by middle schoolers in a classroom setting.
  • Binary Grading: The study focused on a simple "Acceptable/Unacceptable" binary. Real-world education requires more nuanced, multi-point rubrics and qualitative feedback.
  • Language Scope: The study was limited to English and Spanish. Future work should explore other common code-switching pairs (e.g., Hinglish or Taglish) to see if these findings generalize across different language families.

Summary updated

Abstract

Large language models (LLMs) offer promise in generating educational content, providing instructor feedback, and reducing teacher workload on assessments. While prior studies have focused on studying LLM-powered learning analytics, limited research has examined how effective LLMs are in a bilingual context. In this paper, we study the effectiveness of multilingual large language models (MLLMs) across monolingual (English-only, Spanish-only) and bilingual (Spanglish) student writing. We present a learning analytics use case that details LLM performance in assessing acceptable and unacceptable explanations of Science and Social Science concepts. Our findings reveal a significant bias in the grading performance of pre-trained models for bilingual writing compared to English-only and Spanish-only writing. Following this, we fine-tune open-source MLLMs including Llama 3.1 and Mistral NeMo using synthetic datasets generated in English, Spanish, and Spanglish. Our experiments indicate that the models perform significantly better for all three languages after fine-tuning with bilingual data. This study highlights the potential of enhancing MLLM effectiveness to support authentic language practices amongst bilingual learners. It also aims to illustrate the value of incorporating non-English languages into the design and implementation of language models in education.
BibTeX
@misc{syamkumar2024improving,
  author = {Anand Syamkumar and Nora Tseng and Kaycie Barron and Shanglin Yang and Shamya Karumbaiah and Rheeya Uppal and Junjie Hu},
  title = {{Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education}},
  year = {2024},
  eprint = {2411.04308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04308},
}
Figure