A Sentence Judgment System for Grammatical Error Detection

Lung-Hao Lee, Liang-Chih Yu, Kuei-Ching Lee, Yuen-Hsien Tseng, Li-Ping Chang, Hsin-Hsi ChenCOLING 2014

TL;DR

This paper introduces a novel system for detecting grammatical errors in Chinese sentences written by foreign learners. It combines a rule-based linguistic analysis with an n-gram statistical approach to provide learners with feedback on their writing, aiming to improve their understanding of Chinese grammar.

What Problem Does This Paper Address?

The increasing global interest in learning Chinese as a foreign language (CFL) has led to a demand for automated tools to assist learners. While tools for English as a Foreign Language (EFL) are abundant, support for CFL learners, particularly in automatic grammatical error detection and correction, is sparse. Existing tools for Chinese are often primitive, and there's a lack of a common platform for comparing different approaches to CFL grammar checking. This paper aims to fill this gap by developing a system that can identify grammatical errors in Chinese sentences written by CFL learners.

Methods

The proposed system employs a two-pronged approach:

  • Rule-based Linguistic Analysis: This method utilizes 142 rules, handcrafted by linguistic experts, to identify potential rule violations in input sentences. These rules are designed to capture common errors made by CFL learners. The paper details a symbolic notation for representing these rules, including wildcards, exclusions, alternatives, mutual exclusivity, and follow-by conditions.
  • N-gram Statistical Analysis: This approach leverages n-gram (bigram and trigram) language models trained on both correct and incorrect Chinese sentences. The system calculates n-gram scores for input sentences and compares them against models built from correct and incorrect examples. A sentence is flagged as incorrect if its score from the "incorrect" model is higher than that from the "correct" model.

The system also includes a pre-processing step that performs automatic word segmentation and part-of-speech (POS) tagging. For word segmentation, a corpus-based learning method is used to address the "unknown word" problem.

Datasets

  • Training Data:
    • 19,080 Chinese sentences with grammatical errors extracted from the HSK dynamic composition corpus, which were then manually corrected.
  • The Sinica corpus was used to build the n-gram language models.
  • Development Set:
    • "Analysis of 900 Common Erroneous Samples of Chinese Sentences" (Cheng, 1997) was used to handcraft the linguistic rules.
  • Test Set:
    • 880 sentences with grammatical errors generated by CFL learners from the NCKU Chinese Language Center, along with their corresponding 880 manually corrected versions.

Key Insights and Ideas

  • Hybrid Approach Synergy: Combining rule-based and n-gram statistical methods offers complementary strengths. The rule-based approach excels at precision, while the n-gram approach provides better recall.
  • The "AND" Combination: When both rule-based and n-gram methods agree on an error (the "AND" combination), the system achieves significantly higher precision and a lower false positive rate, making it highly reliable for identifying errors.
  • The "OR" Combination: Conversely, using the "OR" combination (flagging an error if either method detects it) dramatically improves recall, ensuring that fewer errors are missed, albeit at the cost of lower precision.
  • Importance of Learner Corpora: The study highlights the challenge of collecting representative CFL learner error data, underscoring the value of the HSK corpus and the NCKU learner data used.
  • Discourse-Oriented Nature of Chinese: The paper notes that the discourse-oriented nature of Chinese, where meaning is often conveyed across clauses, makes syntactic parsing more complex than in English.

Key Results

  • Rule-based: Achieved high precision (0.857) but low recall (0.224) and a low false positive rate (0.038).
  • N-gram (3-gram): Achieved lower precision (0.585) but high recall (0.838) and a high false positive rate (0.595).
  • Rule AND 3-gram: This combined approach yielded the highest precision (0.924) and the lowest false positive rate (0.007), but at the cost of very low recall (0.083).
  • Rule OR 3-gram: This combination achieved perfect recall (1.000) but had very low precision (0.502) and a high false positive rate (0.993).

The results suggest that the choice of method or combination depends on the application's requirements, balancing the need for accuracy versus completeness.

Limitations and Future Work

  • Low Recall for Rule-Based: The rule-based system, while precise, misses a significant number of errors.
  • High False Positives for N-gram: The n-gram approach, while good at finding errors, incorrectly flags many correct sentences.
  • Limited Scope of Rules: The 142 rules, while extensive, may not cover all possible grammatical errors.
  • Future Work: The authors suggest that the system can be integrated into online CFL MOOC platforms for automated assessment. Further research could explore more sophisticated language modeling techniques and a broader set of linguistic rules to improve both precision and recall. The collection of a larger, more representative CFL learner corpus is also a crucial area for future development.

Summary updated

BibTeX
@inproceedings{lee2014sentence,
  author = {Lung-Hao Lee and Liang-Chih Yu and Kuei-Ching Lee and Yuen-Hsien Tseng and Li-Ping Chang and Hsin-Hsi Chen},
  title = {{A Sentence Judgment System for Grammatical Error Detection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2014},
}
Figure