Detection of Non-Native Sentences Using Machine-Translated Training Data

John Lee, Ming Zhou, Xiaohua LiuNAACL 2007

TL;DR

This paper investigates whether machine-translated (MT) sentences can serve as a viable substitute for scarce non-native writing samples in training models for detecting non-native English. The authors find that while MT data is less effective for classifying sentences as native or non-native, it performs comparably to dedicated non-native data for ranking sentence fluency.

What Problem Does This Paper Address?

Providing feedback to non-native English speakers to improve their writing is crucial. While human feedback is ideal, it's not always available or private. Automated systems that can distinguish between native and non-native sentences could offer a scalable solution. However, training such systems typically requires large datasets of non-native writing, which are difficult and expensive to compile. This paper explores a practical alternative: using readily available machine-translated (MT) sentences as training data for these detection tasks.

Methods

The authors frame the problem into two distinct tasks:

  1. Classification: Given a single sentence, determine if it's native or non-native.
  2. Ranking: Given multiple sentences with the same intended meaning, rank them by fluency.

For both tasks, they employ Support Vector Machines (SVMs) using the SVM-Light toolkit. The core innovation lies in comparing the performance of models trained on two different types of data:

  • MT Data: Sentences translated by machine or human from foreign languages (Chinese and Arabic) into English.
  • Non-native Data (JLE): Sentences written by Japanese learners of English, which have been edited by native speakers.

They extract a set of features including:

  • Entropy: From a trigram language model to detect local errors.
  • Parse Score: From a statistical parser, hypothesizing that non-native sentences might have lower scores.
  • Derivations: Analyzing parse tree derivations to catch unusual syntactic structures.
  • DtNoun: Focusing on the head word of noun phrases and their determiners, as article usage is a common error for learners.
  • Colloc: Using an in-house dependency parser to extract word dependencies and identify unusual collocations.

Datasets

The paper uses the following datasets:

  • MT Data:
    • MT train: A large corpus derived from LDC datasets (2002T01, 2003T18, 2006T04) containing over 30,000 sentences.
  • MT dev & MT test: Subsets from LDC2003T17 for development and testing.
  • Non-native Data (JLE):
    • JLE train, JLE dev, & JLE test: Transcripts from Japanese learners of English in a Standard Speaking Test, with false starts and disfluencies cleaned up, and grammatical mistakes tagged. These are smaller than the MT datasets.

Key Insights and Ideas

  • MT data is a mixed bag: While MT sentences are abundant, they don't perfectly mimic the errors and characteristics of human non-native writing.
  • Task dependency is key: The effectiveness of MT data as a substitute is highly dependent on the specific task.
  • Classification is sensitive to data mismatch: The models trained on MT data struggle more with classifying JLE test data, suggesting that the specific error patterns in MT differ significantly from those in human non-native writing.
  • Ranking is more robust to mismatch: For ranking sentences by fluency, MT-trained models perform surprisingly well, even outperforming models trained on a larger non-native corpus. This implies that the relative fluency judgments learned from MT data generalize better.
  • Relative scores matter more for ranking: The SVM's ranking inequalities learn relative scores. As long as a gap exists between native and non-native sentences, these relative judgments can transfer across different types of non-native data.

Key Results

  • Classification:
    • Models trained on the JLE (non-native) training set achieved higher classification accuracy (66.4%) on the JLE test set compared to models trained on the larger MT training set (59.0%).
  • MT-trained SVMs consistently misclassified more native sentences than non-native ones on the JLE test set.
  • Ranking:
    • Models trained on the MT training set achieved higher ranking accuracy (76.2%) on the JLE test set than those trained on the JLE training set (74.6%).
  • This indicates that MT training data can generalize well enough to perform better than a non-native training corpus of size up to 10,000 for the ranking task.

Limitations and Future Work

  • Limitations:
    • The JLE data, while edited by native speakers, originates from spoken transcripts and has a more formal style than spontaneous English, which might not perfectly align with all types of non-native writing.
  • Direct comparison of classification performance with prior work is difficult due to differing corpora.
  • Future Work:
    • Investigate more salient features by studying error-tagged corpora of non-native errors.
  • Explore techniques for combining large MT training corpora with smaller non-native training corpora.
  • The ultimate goal is to identify specific errors in non-native sentences and propose corrections.

Summary updated

BibTeX
@inproceedings{lee2007detection,
  author = {John Lee and Ming Zhou and Xiaohua Liu},
  title = {{Detection of Non-Native Sentences Using Machine-Translated Training Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure