TL;DR
This paper tackles the challenge of automatically assessing spoken English proficiency, especially for second language learners, by leveraging grammatical errors found in written text. They developed an "error feature extractor" (EFEX) that identifies and ranks error types, demonstrating that this approach can improve spoken proficiency assessment, even when models are trained on written data.
What Problem Does This Paper Address?
Assessing spoken language proficiency for English as a Second Language (ESL) learners is crucial but faces significant hurdles. Firstly, there's a scarcity of publicly available, annotated spoken datasets, making it difficult to train robust automatic assessment systems. Secondly, human assessments, while valuable, can be inconsistent and biased, making it hard to translate them into reliable automated scoring. This paper aims to bridge this gap by using grammatical errors, which are consistent indicators of proficiency, as a key feature for automatic assessment, even when the training data is written and the target data is spoken.
Methods
The core of this work is the development of an Error Feature Extractor (EFEX). This model, built using a BERT architecture, takes token embeddings from written text as input and predicts the "biased" estimate of error rates for five distinct error classes: verb tense (VT), lexis and use of words (LUW), prepositions, articles, possessives and part of speech (PAP), agreement (AG), and generic errors (GE). These classes were derived from a larger set of error types by grouping them and ranking their gravity based on their frequency across different proficiency levels in the EFCAMDAT corpus.
The EFEX model is trained on written data (specifically, the EFCAMDAT corpus) where errors are annotated. Once trained, EFEX acts as a feature extractor, providing a vector of error-related features. This vector is then fed into a separate scoring model, also based on BERT, which predicts the overall proficiency score. A key aspect of their approach is the cross-corpora experimentation, where models trained on one dataset (often written) are tested on others, including spoken data. They also explore fine-tuning these models on the target spoken data to further improve performance.
Datasets
The paper utilizes several datasets for training and evaluation:
- EFCAMDAT: This large written corpus, annotated with errors and proficiency levels (mapped to CEFR levels A1-C1), was used to train the EFEX model.
- CLC-FCE (Cambridge Learner Corpus - First Certificate): This corpus of written English exam scripts (around CEFR B2 level) was used for training the scoring models and as a source of written data for cross-corpora experiments.
- ICNALE (International Corpus Network of Asian Learners of English): This dataset contains both written and spoken responses from ESL learners, with CEFR levels assigned. It was used for testing the cross-corpora transferability of the EFEX-enriched models.
- TLT-school: This corpus consists of written and spoken responses from Italian students. It was used to test the EFEX-enriched model's ability to predict spoken proficiency scores, including those derived from Automatic Speech Recognition (ASR) output.
Key Insights and Ideas
- Grammatical errors as a proxy for proficiency: The study strongly suggests that grammatical errors, when analyzed systematically, are reliable indicators of language proficiency and can be effectively used to predict scores, even in spoken language.
- Written-to-spoken transfer learning: The research demonstrates that models trained on written error data can be successfully applied to spoken language assessment, mitigating the scarcity of spoken data. This "transfer learning" approach is a significant contribution.
- EFEX enhances spoken assessment: The EFEX-enriched models consistently showed improvements in predicting spoken proficiency scores, particularly when dealing with ASR output, suggesting that error features can help overcome noise and typical spoken language phenomena.
- Error gravity matters: The paper's approach of ranking error gravity based on frequency across proficiency levels is a novel way to leverage error information, giving more weight to less common errors for a given level.
Key Results
- ICNALE: The EFEX-enriched model showed modest improvements in predicting dialogue scores (PCC) compared to a baseline model when tested on the ICNALE dataset, even without fine-tuning.
- TLT-school: On the TLT-school dataset, the EFEX-enriched model achieved higher performance (PCC of 0.764 on manual transcriptions and 0.642 on ASR output) compared to the baseline, especially after fine-tuning. This suggests that error features are particularly beneficial for spoken language, including ASR-generated text.
- Error distribution and score ranges: The study observed that the EFEX-enriched model's Mean Squared Error (MSE) was slightly higher for very low score ranges (0-3) on the TLT-school dataset. This is attributed to shorter, less varied answers in this range, where EFEX might introduce less relevant information. However, for higher score ranges, EFEX consistently improved performance.
Limitations and Future Work
- Spontaneous speech phenomena: The current EFEX model and experiments removed spontaneous speech phenomena like hesitations and fragments. Future work could integrate error features derived from audio, such as phonological errors and disfluencies, to provide a more comprehensive assessment.
- CEFR levels: The current experiments focused on specific CEFR levels. Future research could investigate the impact of models trained on written data and tested on spoken data across a wider range of CEFR levels.
- Error distinction in spoken language: The authors acknowledge that spoken language has unique error characteristics (e.g., repetitions, preposition insertions that might be acceptable in speech but not writing). Future work could explore distinguishing between errors made by the speaker and those introduced by ASR.
- Beyond errors: For higher proficiency levels, errors alone may not be sufficient for assessment. Future work should consider balancing error-based features with other proficiency indicators for a more holistic evaluation.
- Error remapping and ranking: The paper suggests further exploration of the error feature extractor itself, including rearranging error classes and remapping the error gravity ranking.
Summary updated