TL;DR
This paper explores the feasibility of automatically scoring student answers in a different language than the one the scoring model was trained on. They found that while cross-lingual scoring is possible using machine translation, it comes at a significant cost to performance, and the differences between datasets play a larger role than just language translation errors.
What Problem Does This Paper Address?
Automated content scoring of student answers is a valuable tool, but it's typically limited to a single language. This creates an educational inequality issue: students who understand a subject but struggle with the language of instruction are at a disadvantage. Furthermore, a lack of annotated data in many languages hinders the development of scoring systems for those languages. This paper tackles these issues by investigating whether scoring models trained in one language can be effectively applied to answers in another language, aiming to bridge this gap and address data sparsity.
Methods
The core of the paper's methodology revolves around using state-of-the-art machine translation (MT) to enable cross-lingual scoring. They explore two main strategies:
- Translating Training Data: The original training data in one language is translated into the target language.
- Translating Test Data: The student answers in the target language are translated into the source language of the training data.
They then train and evaluate scoring models using these translated datasets. To isolate the effects of MT, they also perform "double translation" experiments (e.g., English to German, then back to English) to see how MT artifacts impact performance. They use a standard supervised machine learning setup with SVM classifiers and token/character n-grams as features.
Datasets
The primary dataset used is the ASAP (Automated Student Assessment Prize) dataset, a well-known benchmark for short-answer scoring. To enable cross-lingual experiments, the authors created a new dataset called ASAP-DE. This dataset extends the English ASAP corpus by collecting German answers for three specific science prompts. They collected these German answers through crowdsourcing and direct requests.
Key Insights and Ideas
- Machine Translation is a Bridge, But Not a Perfect One: While MT makes cross-lingual scoring possible, it introduces "translationese" – a language style that differs from genuine text and negatively impacts scoring performance.
- Dataset Differences Matter More Than You Think: Surprisingly, the performance drop in cross-lingual scoring is often more attributable to inherent differences between the English and German datasets (e.g., learner populations, writing styles) than to the errors introduced by machine translation itself.
- Spelling Errors Can Be a Double-Edged Sword: MT can sometimes correct common misspellings, which might be beneficial. However, it can also preserve or even introduce new errors, complicating the scoring process.
- Prompt Specificity is Crucial: The degree of performance drop varies significantly across different prompts, suggesting that the nature of the question and the expected answer content play a vital role in cross-lingual transferability.
Key Results
- Performance Drop is Significant: Cross-lingual scoring, even with MT, leads to a considerable decrease in performance compared to monolingual scoring. For example, translating test data from German to English resulted in quadratically weighted kappa (QWK) scores around .26-.35, compared to monolingual German scores of .61-.68.
- Double Translation of Test Data is Detrimental: Translating test data to the training language (e.g., German test data translated to English) and then back to the original language (English) significantly harms performance.
- Double Translation of Training Data is Less Harmful: Surprisingly, double-translating the training data (e.g., English to German, then back to English) had a less pronounced negative impact on performance, and in some cases, even showed slight improvements.
- Monolingual Baselines are Strong: The authors established that their manual scoring of the German dataset was reliable, achieving strong monolingual performance comparable to English baselines.
Limitations and Future Work
- "Translationese" is a Problem: The authors acknowledge that the language produced by machine translation is distinct from natural language and poses a challenge.
- Dataset Differences are a Major Hurdle: The inherent differences between the English and German datasets, likely stemming from different learner populations and educational contexts, are a significant factor limiting cross-lingual performance.
- Future Work: The authors suggest exploring more sophisticated approaches beyond simple MT, such as using cross-lingual word embeddings or neural network architectures designed for cross-lingual transfer. They also aim to extend their experiments to a wider variety of data types and languages.
Summary updated