TL;DR
This paper investigates whether machine-translated (MT) sentences can serve as a viable substitute for scarce non-native writing samples in training models for detecting non-native English. The authors find that while MT data is less effective for classifying sentences as native or non-native, it performs comparably to dedicated non-native data for ranking sentence fluency.
What Problem Does This Paper Address?
Providing feedback to non-native English speakers to improve their writing is crucial. While human feedback is ideal, it's not always available or private. Automated systems that can distinguish between native and non-native sentences could offer a scalable solution. However, training such systems typically requires large datasets of non-native writing, which are difficult and expensive to compile. This paper explores a practical alternative: using readily available machine-translated (MT) sentences as training data for these detection tasks.
Methods
The authors frame the problem into two distinct tasks:
- Classification: Given a single sentence, determine if it's native or non-native.
- Ranking: Given multiple sentences with the same intended meaning, rank them by fluency.
For both tasks, they employ Support Vector Machines (SVMs) using the SVM-Light toolkit. The core innovation lies in comparing the performance of models trained on two different types of data:
- MT Data: Sentences translated by machine or human from foreign languages (Chinese and Arabic) into English.
- Non-native Data (JLE): Sentences written by Japanese learners of English, which have been edited by native speakers.
They extract a set of features including:
- Entropy: From a trigram language model to detect local errors.
- Parse Score: From a statistical parser, hypothesizing that non-native sentences might have lower scores.
- Derivations: Analyzing parse tree derivations to catch unusual syntactic structures.
- DtNoun: Focusing on the head word of noun phrases and their determiners, as article usage is a common error for learners.
- Colloc: Using an in-house dependency parser to extract word dependencies and identify unusual collocations.
Datasets
The paper uses the following datasets:
- MT Data:
- MT train: A large corpus derived from LDC datasets (2002T01, 2003T18, 2006T04) containing over 30,000 sentences.
- MT dev & MT test: Subsets from LDC2003T17 for development and testing.
- Non-native Data (JLE):
- JLE train, JLE dev, & JLE test: Transcripts from Japanese learners of English in a Standard Speaking Test, with false starts and disfluencies cleaned up, and grammatical mistakes tagged. These are smaller than the MT datasets.
Key Insights and Ideas
- MT data is a mixed bag: While MT sentences are abundant, they don't perfectly mimic the errors and characteristics of human non-native writing.
- Task dependency is key: The effectiveness of MT data as a substitute is highly dependent on the specific task.
- Classification is sensitive to data mismatch: The models trained on MT data struggle more with classifying JLE test data, suggesting that the specific error patterns in MT differ significantly from those in human non-native writing.
- Ranking is more robust to mismatch: For ranking sentences by fluency, MT-trained models perform surprisingly well, even outperforming models trained on a larger non-native corpus. This implies that the relative fluency judgments learned from MT data generalize better.
- Relative scores matter more for ranking: The SVM's ranking inequalities learn relative scores. As long as a gap exists between native and non-native sentences, these relative judgments can transfer across different types of non-native data.
Key Results
- Classification:
- Models trained on the JLE (non-native) training set achieved higher classification accuracy (66.4%) on the JLE test set compared to models trained on the larger MT training set (59.0%).
- MT-trained SVMs consistently misclassified more native sentences than non-native ones on the JLE test set.
- Ranking:
- Models trained on the MT training set achieved higher ranking accuracy (76.2%) on the JLE test set than those trained on the JLE training set (74.6%).
- This indicates that MT training data can generalize well enough to perform better than a non-native training corpus of size up to 10,000 for the ranking task.
Limitations and Future Work
- Limitations:
- The JLE data, while edited by native speakers, originates from spoken transcripts and has a more formal style than spontaneous English, which might not perfectly align with all types of non-native writing.
- Direct comparison of classification performance with prior work is difficult due to differing corpora.
- Future Work:
- Investigate more salient features by studying error-tagged corpora of non-native errors.
- Explore techniques for combining large MT training corpora with smaller non-native training corpora.
- The ultimate goal is to identify specific errors in non-native sentences and propose corrections.
Summary updated