From Images to Grounded Edits: A Real-World Benchmark for Chinese Grammatical Error Correction in Handwritten Essays

Weiru Zhang, Yuzhong Hong, Junwei Bao, Lemao Liu, Dejing DouarXiv 2026 · arXiv

TL;DR

The authors introduce Grounded Chinese Grammatical Error Correction (G-CGEC), a new task that requires AI to not only fix grammatical errors in handwritten essays but also pinpoint their exact locations on the original image. To support this, they release G-CGEC-Eval, a real-world benchmark of K-12 essays, and demonstrate that while current vision-language models struggle with end-to-end correction, multi-agent workflows that coordinate specialized tools offer the most promising path forward.

What Problem Does This Paper Address?

Traditional Chinese Grammatical Error Correction (CGEC) is usually treated as a "text-to-text" problem, similar to machine translation. While this works for digital text, it fails in the real world of K-12 education where students write by hand.

Standard approaches often rely on a "OCR + Text Correction" pipeline. This creates a cascading error effect: if the OCR misreads a messy character, the correction model receives garbled input and fails. Furthermore, for an AI assistant to be truly useful to a teacher, it needs to highlight errors directly on the student's paper, not just provide a corrected transcript. Existing benchmarks lack the visual diversity (lighting, shadows, messy handwriting) and the "grounding" (bounding boxes) necessary to develop these practical tools.

Figure 1: When the input is in the image format, our image-to- grounded-edit formulation of the error correction task offers intuitive advantages: giving a system complete information as opposed to lossy OCR transcription; providing practical usefulness (e.g., for K-12 education).

Methods

The researchers evaluated several Vision-Language Models (VLMs)—including GPT-4o/5.5, MiMo-V2.5, and Qwen—using three distinct inference strategies:

  1. Single-pass Scheme: An end-to-end approach where the VLM attempts to output grounded edits (text + coordinates) directly from the image in one go.
  2. Pipeline Scheme: A modular, sequential flow where a dedicated OCR module transcribes the text, a VLM performs the correction, and a grounding module maps the edits back to the image.
  3. Agentic Scheme: A flexible approach where a "master" VLM dynamically coordinates with auxiliary OCR and grounding tools. They tested both single-agent and multi-agent workflows, where sub-agents specialize in specific error types (e.g., punctuation vs. grammar).

To measure success, they used standard Intersection over Union (IoU) and proposed a more tolerant Center-Distance (CD) metric. CD measures the distance between the centers of predicted and actual error boxes, which is more helpful for evaluating models that are "close" but don't perfectly overlap with ground truth.

Datasets

The paper introduces G-CGEC-Eval, a benchmark consisting of 150 real-world K-12 handwritten essays (251 images).

  • Scale: It contains 2,913 annotated edits.
  • Complexity: Unlike previous datasets, it features "in-the-wild" noise like perspective distortion and uneven lighting.
  • Fine-grained Labels: Errors are classified into 7 coarse categories and 26 fine-grained subtypes. Syntactic, discourse-level, and punctuation errors account for over 80% of the total.
Figure 2: Error category distribution in the G-CGEC-Eval benchmark. Horizontal bars show fine-grained categories grouped by 7 coarse categories (bold labels, separated by gray lines). All bars within a group share the same color. Note that syntactic, discourse-level errors, and punctuation errors together account for over 80% of all errors.

Key Insights and Ideas

  • The "Unified" Model Gap: Even the most powerful proprietary VLMs perform surprisingly poorly in the single-pass setting (F1 scores < 1%). This indicates that current models struggle to simultaneously "see" dense, messy text and "reason" about its grammar.
  • Coupled Failures: The authors found that visual grounding and linguistic reasoning failures are deeply linked. As shown in Figure 3, the majority of errors involve a model failing at both locating the character and identifying the correct edit.
  • OCR is Still a Bottleneck: While modular pipelines help, they are limited by OCR quality. An "oracle" experiment using human-perfect transcripts showed a massive performance jump, proving that better handwriting recognition is still a critical prerequisite for this task.
  • Multi-Agent Advantage: Decomposing the task by error type (e.g., specialized agents for spelling vs. syntax) yielded the strongest results. This "divide and conquer" strategy helps the system discover more candidate errors than a single-agent approach.

Key Results

  • Top Performer: The Error-type-based multi-agent workflow using GPT-5.5 achieved the highest F1 score of 22.83% (at [email protected]).
  • Modular vs. End-to-End: Modular approaches (pipeline and agentic) significantly outperformed single-pass models. For example, MiMo-V2.5 jumped from an F1 of 0.18% (single-pass) to 14.01% (pipeline).
  • The Grounding Ceiling: Even when the spatial requirement was completely relaxed (using the Center-Distance metric), F1 scores remained below 40%. This suggests that even if we "solve" the vision part, the linguistic challenge of correcting handwritten Chinese remains formidable.
Figure 3: Fine-grained failure decomposition. Each configuration has three bars: TP, FP, and FN. Bar heights are raw counts. FP and FN bars are internally stacked by subtype, and in-bar percentages (rounded to integers) denote subtype proportions. In the legend, visual, textual, and both indicate which kind of match failed.

Limitations and Future Work

  • Efficiency: The high-performing agentic workflows are computationally expensive and slow due to their multi-step nature. Developing more efficient, perhaps distilled, models is a key next step.
  • Need for Fine-Tuning: This study focused on zero-shot performance. The authors anticipate that task-specific fine-tuning on grounded datasets will be necessary to reach human-level utility.
  • Integrated Reasoning: The primary bottlenecks remain character-level grounding and linguistic error discovery. Future research should focus on models that more tightly integrate spatial perception with contextual grammatical understanding.

Summary updated

Abstract

Chinese Grammatical Error Correction (CGEC) is typically formulated as a text-to-text problem. While suitable for digital texts, this formulation falls short in real-world K-12 education, where AI-assisted corrections are expected directly on essay images, which are primarily handwritten and exhibit many irregularities. To bridge this gap, we introduce Grounded Chinese Grammatical Error Correction (G-CGEC), an image-to-grounded-edit task requiring systems to predict both corrections and their positions on essay images. We propose G-CGEC-Eval, a benchmark comprising real-world images of K-12 Chinese essays with fine-grained annotations. For evaluation, we adapt IoU-based metrics from object detection and propose a more tolerant alternative based on center-distance matching. Using this benchmark, we evaluate open-source and proprietary vision-language models under a direct single-pass setting, as well as settings with auxiliary components (OCR and character grounding) coordinated via pipelining or agentic harnessing. Results show that even top-tier proprietary models perform surprisingly poorly in the single-pass setting, while pipelining and agentic harnessing yield meaningful improvements. Notably, an error-type-based multi-agent workflow achieves the best performance, establishing it as the strongest baseline for future G-CGEC research. Through ablations and fine-grained analysis, we find the task remains far from solved, with linguistic error discovery and character-level grounding as the primary bottlenecks. This underscores the need for models that better integrate visual perception with linguistic reasoning and visual grounding.
BibTeX
@misc{zhang2026from,
  author = {Weiru Zhang and Yuzhong Hong and Junwei Bao and Lemao Liu and Dejing Dou},
  title = {{From Images to Grounded Edits: A Real-World Benchmark for Chinese Grammatical Error Correction in Handwritten Essays}},
  year = {2026},
  eprint = {2611.17966},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17966},
}
Figure