Abstract
Language Development Assessment (LDA) is a critical task for tracking the linguistic progress of second-language (L2) learners by comparing texts written at different stages of their acquisition journey. Despite its importance, LDA research is often hindered by data scarcity in low-resource languages and a reliance on labor-intensive, language-specific feature engineering. This paper introduces IndoCL, the first longitudinal benchmark dataset for Indonesian LDA. We also propose a novel, language-independent neural architecture that captures both the sequential differences and shared linguistic traits between essay pairs. Our model achieves a state-of-the-art accuracy of 91.61% on IndoCL, significantly outperforming standard pre-trained models and Large Language Models like ChatGPT.
Introduction
Tracking a learner's progress over time is essential for effective language instruction. In the context of automated assessment, this is framed as identifying the chronological order of two texts written by the same individual. If a model can correctly determine that "Text B" represents a more advanced stage of development than "Text A," it demonstrates an understanding of the markers of linguistic improvement.
While LDA is well-established for high-resource languages like English, it faces two major hurdles in other contexts: the lack of longitudinal corpora and the methodological rigidity of traditional approaches. Most existing systems rely on manual feature extraction (e.g., counting specific grammatical structures), which is difficult to transfer across languages. Furthermore, current deep learning approaches often struggle to effectively model the complex interactions between related texts.
The IndoCL Dataset
To address the data gap, we constructed IndoCL, a longitudinal corpus consisting of 212 essays written by 35 undergraduate students majoring in Indonesian. These students, all native Chinese speakers, were tracked over a three-year period. To ensure the robustness of the benchmark, we also evaluated our approach on two established datasets: CItA (L1 Italian) and COWS-L2H (L2 Spanish).
Methodology
We propose a language-independent neural pipeline designed to capture the nuances of language acquisition without relying on hard-coded grammar rules. The architecture consists of four primary modules:
- Text Representation: Utilizes a pre-trained BERT model to generate semantic embeddings for the essay pairs.
- Sequential Information Attention: This module employs an attention mechanism to capture how the first-written essay relates to the second, specifically highlighting differences in competence.
- Similarity Representation Learning (SRL): Moving beyond simple distance metrics, this module learns an abstract vector-based similarity. It focuses on the common information shared between texts, providing a stable baseline for comparison.
- Assessment Module: The final classification is performed by a Feed-Forward Network (FFN) that concatenates raw representations, attention-enhanced features, and the similarity vector.
Results and Discussion
Our proposed model demonstrated superior performance across all evaluated languages. On the IndoCL dataset, it achieved 91.61% accuracy, outperforming the BERT baseline (88.87%) and the previous state-of-the-art SIAM model (89.84%). Across the three-language average (Indonesian, Italian, Spanish), the model maintained a high accuracy of 82.66%.
The Role of Similarity
Ablation studies confirmed the importance of the Similarity Representation Learning (SRL) module. Removing it resulted in a 1.61% drop in accuracy, supporting our hypothesis that modeling what remains stable in a learner's writing is just as critical as modeling what changes.
Benchmarking LLMs
We also benchmarked ChatGPT (GPT-3.5) using zero-shot prompting. Interestingly, the LLM performed poorly, achieving only ~56% accuracy on the Indonesian task. This suggests a significant gap in the ability of current general-purpose LLMs to grasp the subtle nuances of language acquisition in low-resource contexts compared to specialized neural models.
Conclusion
IndoCL provides a much-needed benchmark for Indonesian Language Development Assessment. By combining sequential attention with similarity representation learning, our model offers a robust, language-independent solution for tracking learner progress. Future work will focus on expanding the IndoCL corpus to include more diverse learner backgrounds and exploring the impact of fine-tuning Large Language Models specifically for LDA tasks.
Summary updated