Abstract
Grammatical error correction (GEC) system is a practical task used in the real world, showing high achievements alongside the development of large language models (LLMs). However, these achievements have been primarily obtained in English, and there is a relative lack of performance for non-English data, such as Korean. We hypothesize that this insufficiency occurs because relying solely on the parametric knowledge of LLMs makes it difficult to thoroughly understand the given context in the Korean GEC. Therefore, we propose a Knowledge-Augmented GEC (KAGEC) framework that incorporates evidential information from external sources into the prompt for the GEC task. KAGEC first extracts salient phrases from the given source and retrieves non-parametric knowledge based on these phrases, aiming to enhance the context-aware generation capabilities of LLMs. Furthermore, we conduct validations for fine-grained error types to identify those requiring a retrieval-augmented manner when LLMs perform Korean GEC. According to experimental results, most LLMs, including ChatGPT, demonstrate significant performance improvements when applying KAGEC.
Introduction
Large Language Models (LLMs) have revolutionized English Grammatical Error Correction (GEC), yet their performance on morphologically rich languages like Korean remains suboptimal. A primary issue identified in current research is "under-correction," where models fail to identify errors that are syntactically plausible but semantically nonsensical. This stems from a reliance on internal "parametric" knowledge, which often lacks the specific semantic grounding needed for complex linguistic nuances in lower-resource settings.
The KAGEC Framework
To bridge this gap, we introduce Knowledge-Augmented GEC (KAGEC), a framework that integrates external non-parametric knowledge into the GEC process. Unlike conventional GEC, which feeds a task instruction and source sentence directly to an LLM, KAGEC employs a multi-step retrieval-augmented generation (RAG) pipeline:
- Salient Phrase Extraction: The model identifies key phrases from the erroneous input.
- Knowledge Retrieval: These phrases are used to query the National Institute of Korean Language’s Standard Korean Dictionary.
- Augmented Input: The original sentence is combined with retrieved definitions to provide explicit semantic context.
- Correction Generation: The LLM generates the final output based on this enriched prompt.
Experimental Methodology
The framework was evaluated using the K-NCT dataset, a benchmark of 3,000 Korean sentence pairs. Errors were categorized into three groups based on difficulty. Preliminary results indicated that while LLMs handle Group 1 errors (e.g., spacing and punctuation) effectively, they struggle with Group 3 errors: Element (sentence components), Auxiliary Predicate (verb usage), and Behavioral (semantic impossibilities).
Results and Discussion
Experimental results across ChatGPT, Gemini, and Llama2-70B demonstrate that KAGEC significantly improves performance in the most challenging error categories.
As shown in the table above, ChatGPT's performance on "Element" errors improved by over 10 BLEU points (from 47.57 to 58.11). Notably, Llama2-70B achieved a massive leap in "Auxiliary Predicate" correction, with BLEU rising from 52.17 to 64.54.
The KAGEC framework's success highlights the "Search if you don't know" philosophy. The study found that providing semantic definitions of the specific words in a sentence is far more effective than providing random knowledge or general error type descriptions. This suggests that the model's bottleneck in Korean GEC is not a lack of understanding of grammar rules, but a lack of specific semantic context.
Furthermore, the comparison with other prompting techniques like Chain-of-Thought (CoT) and Decomposed prompting shows that KAGEC is uniquely suited for this task. While CoT often leads to hallucinations or over-simplification in GEC, KAGEC grounds the model's reasoning in factual definitions, leading to more precise corrections. However, a "complexity ceiling" was observed with smaller models like Llama2-13B, which occasionally showed performance degradation, suggesting that the ability to utilize external knowledge effectively scales with model capacity.
Conclusion
KAGEC proves that grounding LLMs with authoritative external definitions is a powerful strategy for non-English GEC. By providing semantic context that internal weights may lack, KAGEC reduces hallucinations and improves correction accuracy. Future research will focus on optimizing the latency of the retrieval pipeline and extending the framework to other low-resource languages.
Summary updated