Modeling Chinese L2 Writing Development: The LLM-Surprisal Perspective

Jingying Hu, Yan CongNAACL 2025

TL;DR

This paper explores using "LLM-surprisal" – a measure of how unexpected a word is given its context, as predicted by large language models (LLMs) – to assess second language (L2) writing proficiency in Chinese. They found that LLM-surprisal effectively differentiates proficiency levels and that a specialized Chinese LLM significantly outperforms general and multilingual models, suggesting the importance of domain-specific training for low-resource languages.

What Problem Does This Paper Address?

Assessing L2 writing proficiency is crucial for language learning and education, but traditional methods can be time-consuming and subjective. While computational approaches are emerging, their cross-linguistic applicability and interpretability remain underexplored, especially for typologically distant languages like Chinese. Specifically, this paper investigates whether LLM-surprisal, a measure that quantifies linguistic unpredictability, can serve as a robust and interpretable metric for modeling Chinese L2 writing development, and how different LLM architectures and training data affect its performance in this context.

Methods

The core of the study involves calculating LLM-surprisal for essays written by Chinese L2 learners. LLM-surprisal is computed as the negative log-probability of a word (or character in this case) given its preceding context, as estimated by an LLM. The researchers used three transformer-based LLMs:

  • Bloom: A large-scale multilingual model.
  • Chinese-LLaMA: A model pre-trained on both simplified and traditional Chinese.
  • Taiwan-LLM: An LLM specifically tailored for Traditional Chinese, with an emphasis on Taiwanese linguistic norms.

They then correlated these LLM-surprisal scores with 16 classic linguistic complexity indices (e.g., character sophistication, lexical diversity, syntactic complexity, discourse coherence) to understand what aspects of writing quality LLM-surprisal captures.

Datasets

The study utilized the publicly available TOCFL Learner dataset. This dataset contains written essays from learners of varying L1 backgrounds, covering CEFR proficiency levels A2 to C1. For their analysis, the researchers selected 65 essays from each CEFR level, ensuring balanced comparisons across proficiency groups and L1 backgrounds. Essays were also matched for length (approximately 200 Chinese characters) to control for this variable's influence on surprisal.

Key Insights and Ideas

  • LLM-surprisal is a universal metric: The study confirms that LLM-surprisal can effectively differentiate proficiency levels in Chinese L2 writing, mirroring findings in English L2 writing. This suggests its cross-linguistic robustness.
  • Specialized LLMs matter for low-resource languages: The Taiwan-LLM, trained specifically on Traditional Chinese, significantly outperformed the multilingual Bloom and the general Chinese-LLaMA. This highlights the critical role of domain-specific training data for effective modeling in low-resource language settings.
  • LLM-surprisal captures multifaceted complexity: Beyond just lexical and syntactic features, LLM-surprisal, especially when computed by the Taiwan-LLM, showed significant correlations with a broad range of linguistic complexity indices, including character complexity and discourse coherence (e.g., use of first-person pronouns in a pro-drop language like Chinese).
  • Interpretability is key: The strong correlations with classic indices suggest that LLM-surprisal isn't just a black box but reflects meaningful linguistic properties relevant to L2 proficiency.

Key Results

  • All three LLMs showed significant differences in LLM-surprisal scores across proficiency levels, with advanced learners (C1) consistently exhibiting the lowest surprisal.
  • The Taiwan-LLM demonstrated the largest effect size and the most distinct separation between adjacent proficiency levels, indicating its superior sensitivity.
  • LLM-surprisal scores were negatively correlated with higher proficiency levels across all models, meaning more proficient writers produced less "surprising" text.
  • Taiwan-LLM surprisal scores showed significant correlations with a wide range of classic complexity indices, including character levels, lexical semantic diversity, and various syntactic measures.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited dataset size: The number of essays per proficiency level was limited, and writing genres were not strictly controlled due to the nature of the TOCFL test.
  • Genre variability: Different genres across proficiency levels might confound the relationship between LLM-surprisal and proficiency.

Future work could involve:

  • Analyzing essays from a single genre across multiple proficiency levels.
  • Using larger, more balanced datasets.
  • Investigating the impact of genre on LLM-surprisal.
  • Exploring LLM-surprisal in spoken or multimodal L2 contexts.
  • Further disentangling the effects of language exposure, model architecture, and data familiarity in surprisal-based assessments.

Summary updated

BibTeX
@inproceedings{hu2025modeling,
  author = {Jingying Hu and Yan Cong},
  title = {{Modeling Chinese L2 Writing Development: The LLM-Surprisal Perspective}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure