IndoCL: Benchmarking Indonesian Language Development Assessment

Nankai Lin, Hongyan Wu, Weixiong Zheng, Xingming Liao, Shengyi Jiang, Aimin Yang, Lixian XiaoEMNLP 2024 · DOI

Abstract

Language Development Assessment (LDA) is a critical task for tracking the linguistic progress of second-language (L2) learners by comparing texts written at different stages of their acquisition journey. Despite its importance, LDA research is often hindered by data scarcity in low-resource languages and a reliance on labor-intensive, language-specific feature engineering. This paper introduces IndoCL, the first longitudinal benchmark dataset for Indonesian LDA. We also propose a novel, language-independent neural architecture that captures both the sequential differences and shared linguistic traits between essay pairs. Our model achieves a state-of-the-art accuracy of 91.61% on IndoCL, significantly outperforming standard pre-trained models and Large Language Models like ChatGPT.

Introduction

Tracking a learner's progress over time is essential for effective language instruction. In the context of automated assessment, this is framed as identifying the chronological order of two texts written by the same individual. If a model can correctly determine that "Text B" represents a more advanced stage of development than "Text A," it demonstrates an understanding of the markers of linguistic improvement.

Overview of the LDA task: identifying the chronological order of learner texts to assess writing proficiency.

While LDA is well-established for high-resource languages like English, it faces two major hurdles in other contexts: the lack of longitudinal corpora and the methodological rigidity of traditional approaches. Most existing systems rely on manual feature extraction (e.g., counting specific grammatical structures), which is difficult to transfer across languages. Furthermore, current deep learning approaches often struggle to effectively model the complex interactions between related texts.

The IndoCL Dataset

To address the data gap, we constructed IndoCL, a longitudinal corpus consisting of 212 essays written by 35 undergraduate students majoring in Indonesian. These students, all native Chinese speakers, were tracked over a three-year period. To ensure the robustness of the benchmark, we also evaluated our approach on two established datasets: CItA (L1 Italian) and COWS-L2H (L2 Spanish).

Methodology

We propose a language-independent neural pipeline designed to capture the nuances of language acquisition without relying on hard-coded grammar rules. The architecture consists of four primary modules:

  1. Text Representation: Utilizes a pre-trained BERT model to generate semantic embeddings for the essay pairs.
  2. Sequential Information Attention: This module employs an attention mechanism to capture how the first-written essay relates to the second, specifically highlighting differences in competence.
  3. Similarity Representation Learning (SRL): Moving beyond simple distance metrics, this module learns an abstract vector-based similarity. It focuses on the common information shared between texts, providing a stable baseline for comparison.
  4. Assessment Module: The final classification is performed by a Feed-Forward Network (FFN) that concatenates raw representations, attention-enhanced features, and the similarity vector.
The proposed neural architecture for Language Development Assessment, featuring Sequential Information Attention and Similarity Representation Learning.

Results and Discussion

Our proposed model demonstrated superior performance across all evaluated languages. On the IndoCL dataset, it achieved 91.61% accuracy, outperforming the BERT baseline (88.87%) and the previous state-of-the-art SIAM model (89.84%). Across the three-language average (Indonesian, Italian, Spanish), the model maintained a high accuracy of 82.66%.

The Role of Similarity

Ablation studies confirmed the importance of the Similarity Representation Learning (SRL) module. Removing it resulted in a 1.61% drop in accuracy, supporting our hypothesis that modeling what remains stable in a learner's writing is just as critical as modeling what changes.

Benchmarking LLMs

We also benchmarked ChatGPT (GPT-3.5) using zero-shot prompting. Interestingly, the LLM performed poorly, achieving only ~56% accuracy on the Indonesian task. This suggests a significant gap in the ability of current general-purpose LLMs to grasp the subtle nuances of language acquisition in low-resource contexts compared to specialized neural models.

Conclusion

IndoCL provides a much-needed benchmark for Indonesian Language Development Assessment. By combining sequential attention with similarity representation learning, our model offers a robust, language-independent solution for tracking learner progress. Future work will focus on expanding the IndoCL corpus to include more diverse learner backgrounds and exploring the impact of fine-tuning Large Language Models specifically for LDA tasks.

Summary updated

BibTeX
@inproceedings{lin2024indocl,
  author = {Nankai Lin and Hongyan Wu and Weixiong Zheng and Xingming Liao and Shengyi Jiang and Aimin Yang and Lixian Xiao},
  title = {{IndoCL: Benchmarking Indonesian Language Development Assessment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.280},
}
Figure