Search if you don't know! Knowledge-Augmented Korean Grammatical Error Correction with Large Language Models

Seonmin Koo, Jinsung Kim, Chanjun Park, Heuiseok LimEMNLP 2024 · OpenReview

Abstract

Grammatical error correction (GEC) system is a practical task used in the real world, showing high achievements alongside the development of large language models (LLMs). However, these achievements have been primarily obtained in English, and there is a relative lack of performance for non-English data, such as Korean. We hypothesize that this insufficiency occurs because relying solely on the parametric knowledge of LLMs makes it difficult to thoroughly understand the given context in the Korean GEC. Therefore, we propose a Knowledge-Augmented GEC (KAGEC) framework that incorporates evidential information from external sources into the prompt for the GEC task. KAGEC first extracts salient phrases from the given source and retrieves non-parametric knowledge based on these phrases, aiming to enhance the context-aware generation capabilities of LLMs. Furthermore, we conduct validations for fine-grained error types to identify those requiring a retrieval-augmented manner when LLMs perform Korean GEC. According to experimental results, most LLMs, including ChatGPT, demonstrate significant performance improvements when applying KAGEC.

Introduction

Large Language Models (LLMs) have revolutionized English Grammatical Error Correction (GEC), yet their performance on morphologically rich languages like Korean remains suboptimal. A primary issue identified in current research is "under-correction," where models fail to identify errors that are syntactically plausible but semantically nonsensical. This stems from a reliance on internal "parametric" knowledge, which often lacks the specific semantic grounding needed for complex linguistic nuances in lower-resource settings.

The KAGEC Framework

To bridge this gap, we introduce Knowledge-Augmented GEC (KAGEC), a framework that integrates external non-parametric knowledge into the GEC process. Unlike conventional GEC, which feeds a task instruction and source sentence directly to an LLM, KAGEC employs a multi-step retrieval-augmented generation (RAG) pipeline:

  1. Salient Phrase Extraction: The model identifies key phrases from the erroneous input.
  2. Knowledge Retrieval: These phrases are used to query the National Institute of Korean Language’s Standard Korean Dictionary.
  3. Augmented Input: The original sentence is combined with retrieved definitions to provide explicit semantic context.
  4. Correction Generation: The LLM generates the final output based on this enriched prompt.
Comparison between conventional GEC and the proposed KAGEC framework.

Experimental Methodology

The framework was evaluated using the K-NCT dataset, a benchmark of 3,000 Korean sentence pairs. Errors were categorized into three groups based on difficulty. Preliminary results indicated that while LLMs handle Group 1 errors (e.g., spacing and punctuation) effectively, they struggle with Group 3 errors: Element (sentence components), Auxiliary Predicate (verb usage), and Behavioral (semantic impossibilities).

Results and Discussion

Experimental results across ChatGPT, Gemini, and Llama2-70B demonstrate that KAGEC significantly improves performance in the most challenging error categories.

Performance comparison of KAGEC against baseline, CoT, and Decomposed prompting across different LLMs for Group 3 error types.

As shown in the table above, ChatGPT's performance on "Element" errors improved by over 10 BLEU points (from 47.57 to 58.11). Notably, Llama2-70B achieved a massive leap in "Auxiliary Predicate" correction, with BLEU rising from 52.17 to 64.54.

The KAGEC framework's success highlights the "Search if you don't know" philosophy. The study found that providing semantic definitions of the specific words in a sentence is far more effective than providing random knowledge or general error type descriptions. This suggests that the model's bottleneck in Korean GEC is not a lack of understanding of grammar rules, but a lack of specific semantic context.

Furthermore, the comparison with other prompting techniques like Chain-of-Thought (CoT) and Decomposed prompting shows that KAGEC is uniquely suited for this task. While CoT often leads to hallucinations or over-simplification in GEC, KAGEC grounds the model's reasoning in factual definitions, leading to more precise corrections. However, a "complexity ceiling" was observed with smaller models like Llama2-13B, which occasionally showed performance degradation, suggesting that the ability to utilize external knowledge effectively scales with model capacity.

Conclusion

KAGEC proves that grounding LLMs with authoritative external definitions is a powerful strategy for non-English GEC. By providing semantic context that internal weights may lack, KAGEC reduces hallucinations and improves correction accuracy. Future research will focus on optimizing the latency of the retrieval pipeline and extending the framework to other low-resource languages.

Summary updated

Abstract

Grammatical error correction (GEC) system is a practical task used in the real world, showing high achievements alongside the development of large language models (LLMs). However, these achievements have been primarily obtained in English, and there is a relative lack of performance for non-English data, such as Korean. We hypothesize that this insufficiency occurs because relying solely on the parametric knowledge of LLMs makes it difficult to thoroughly understand the given context in the Korean GEC. Therefore, we propose a Knowledge-Augmented GEC (KAGEC) framework that incorporates evidential information from external sources into the prompt for the GEC task. KAGEC first extracts salient phrases from the given source and retrieves non-parametric knowledge based on these phrases, aiming to enhance the context-aware generation capabilities of LLMs. Furthermore, we conduct validations for fine-grained error types to identify those requiring a retrieval-augmented manner when LLMs perform Korean GEC. According to experimental results, most LLMs, including ChatGPT, demonstrate significant performance improvements when applying KAGEC.
BibTeX
@inproceedings{koo2024search,
  author = {Seonmin Koo and Jinsung Kim and Chanjun Park and Heuiseok Lim},
  title = {{Search if you don't know! Knowledge-Augmented Korean Grammatical Error Correction with Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=KjCd8Wo8sK},
}
Figure