TL;DR
This paper tackles the challenge of automated essay scoring (AES) in low-resource languages by proposing a novel zero-shot cross-lingual approach. By leveraging multilingual pre-trained language models and a contrastive learning strategy to align writing quality across languages, they achieve state-of-the-art performance on unseen languages.
What Problem Does This Paper Address?
Automated Essay Scoring (AES) aims to automatically evaluate the quality of written essays. A common approach involves training models on large datasets of human-rated essays. However, creating high-quality, labeled datasets is expensive and time-consuming, leading to a scarcity of data for many languages. This "low-resource problem" is a significant hurdle for extending AES beyond well-resourced languages like English. While cross-lingual AES (training on one language and testing on another) offers a solution, existing methods often rely on hand-crafted features or machine translation, which have limitations like language-specific tool dependencies, incomplete feature representation, and potential alteration of essay content. This paper addresses the need for a robust, zero-shot cross-lingual AES method that can effectively learn and transfer scoring knowledge across languages without requiring target-language training data.
Methods
The core of the proposed method lies in two key components:
- Multilingual Pretrained Language Models (PLMs) as Essay Encoder: Instead of traditional feature engineering or monolingual embeddings, the authors utilize powerful multilingual PLMs (like mBERT and XLM-R) as the encoder. These models are pre-trained on massive multilingual corpora, enabling them to deeply and comprehensively represent essays in various languages using a unified subword vocabulary. This addresses the challenge of effectively representing multilingual essays.
- Writing Quality Alignment via Contrastive Learning: To learn language-independent scoring knowledge, the authors introduce a novel contrastive learning strategy. They define three levels of writing quality (low, medium, high) based on essay scores. The contrastive learning objective aims to pull essay representations of the same writing quality closer together in an embedding space, while pushing apart representations of different writing qualities. This "aligns" writing quality across languages, allowing the model to capture transferable scoring knowledge. Essays with identical quality, regardless of their original language, are brought closer, facilitating the learning of language-agnostic quality assessment.
Datasets
The authors evaluate their method on four datasets covering six languages:
- MERLIN: Czech, German, Italian
- Write & Improve: English
- CEDEL2: Spanish
- COPLE2: Portuguese
These datasets consist of essays written by second-language learners and are manually scored using CEFR levels or placement test scores, which are then mapped to a 1-6 scale for consistency. The paper notes that essays in MERLIN and COPLE2 that were unrated or spoken were discarded, and only written essays were used.
Key Insights and Ideas
- Scoring Knowledge is Transferable: The fundamental assumption is that certain aspects of writing quality evaluation (e.g., essay structure, coherence, lexical diversity) are universal and can be learned and transferred across languages, despite linguistic differences.
- Multilingual PLMs are Powerful Encoders: The use of pre-trained multilingual models provides a strong foundation for understanding and representing essays from diverse linguistic backgrounds in a unified manner.
- Contrastive Learning for Alignment: Aligning writing quality through contrastive learning is a crucial step in bridging the gap between languages and enabling zero-shot transfer. It effectively teaches the model to recognize quality independent of language.
- Granularity Matters: The study found that a coarser granularity of writing quality (e.g., 3 levels) leads to better performance than a finer-grained scale (e.g., 6 levels), suggesting that too much detail can hinder the model's ability to distinguish quality effectively in a contrastive setting.
- Source Language Diversity is Beneficial: Increasing the number of source languages for training improves both the average performance and the consistency of cross-lingual scoring.
Key Results
- The proposed method, particularly "mBERT-uncased + XCL," achieved state-of-the-art zero-shot cross-lingual scoring performance, outperforming both statistical and neural baselines.
- The average Quadratic Weighted Kappa (QWK) for their best model was 0.568, which was significantly higher than the best baseline (0.548) and a neural baseline (0.481).
- Their model achieved the highest QWK scores on German (0.789), English (0.438), and Spanish (0.565).
- The writing quality alignment component (XCL) provided a performance boost, improving the QWK from 0.547 to 0.568 for the mBERT-uncased model.
- The impact of language similarity between source and target languages was found to be limited, suggesting that AES quality evaluation involves more than just linguistic similarity.
Limitations and Future Work
- Data Constraints: The study was limited to Indo-European languages due to data availability. Future work aims to include more diverse language families.
- Finer-Grained Scoring Perspectives: The authors suggest that aligning finer-grained scoring perspectives beyond overall writing quality, such as structure completeness, could further improve performance.
- Model Re-training: While their method avoids re-training for new target languages, some baseline methods (like essay translation) require it, making them less efficient. The proposed method's efficiency in this regard is a key advantage.
Summary updated