Reference-less Measure of Faithfulness for Grammatical Error Correction

Leshem Choshen, Omri Abend

Abstract

Evaluating Grammatical Error Correction (GEC) systems is a complex task due to the inherent variability in valid linguistic corrections. Traditional reference-based metrics often penalize correct but non-matching outputs. This paper introduces USIM (Universal Conceptual Cognitive Annotation Similarity), a reference-less metric designed to evaluate the "faithfulness" of a correction to the original source meaning. By utilizing semantic graph matching, USIM provides a robust measure of meaning preservation without the need for human-written gold standards.

Introduction

In the field of Grammatical Error Correction (GEC), the standard evaluation paradigm involves comparing system outputs against human-authored reference sentences. However, this approach is limited by the "one-to-many" problem: a single ungrammatical sentence can often be corrected in multiple, equally valid ways. If a system produces a high-quality correction that deviates from the specific wording of the reference, it is unfairly penalized.

To address this, researchers have turned to reference-less measures (RLMs). While existing RLMs effectively assess fluency (grammaticality), they often fail to measure faithfulness (meaning preservation). A system might achieve high fluency by simply deleting problematic phrases or changing the subject, which results in a poor correction. USIM fills this gap by focusing on semantic stability between the source and the corrected text.

Methods: The USIM Metric

USIM leverages Universal Conceptual Cognitive Annotation (UCCA) to represent sentences. Unlike syntactic representations that rely on parts of speech, UCCA focuses on the underlying conceptual structure, mapping sentences into graphs of "Scenes" and "Participants." This focus on semantics makes it particularly resilient to the broken syntax typical of learner language.

UCCA graph alignment between a source sentence and its correction.

The USIM score is calculated through a multi-step graph alignment process:

  1. Leaf Alignment: Words (leaves) in the source and correction graphs are aligned based on edit distance.
  2. Node Propagation: Alignment is propagated to internal nodes based on the overlap of their respective yields (the words they cover).
  3. F-Score Calculation: The final metric is an F-score derived from the matching semantic edges between the two aligned graphs. A high score indicates that the core meaning has been preserved despite grammatical changes.

Experimental Setup and Datasets

The authors validated USIM using several key datasets:

  • NUCLE: A corpus of learner English used for primary experiments, with a subset annotated with UCCA structures.
  • JFLEG: Used to test sensitivity to unfaithful corrections by evaluating systems of varying quality.
  • Wikipedia: Used to train the TUPA parser, which allows for the automated generation of UCCA graphs, enabling USIM to function as a fully automated metric.

Key Results and Discussion

The experimental results demonstrate that USIM is a reliable proxy for human judgment of faithfulness. As shown in the validation data, valid corrections achieved a USIM score of 0.84, which aligns closely with the human Inter-Annotator Agreement (IAA) score of 0.83.

Comparison of USIM scores and Inter-Annotator Agreement (IAA) across different sentence pairs.

Key findings include:

  • Sensitivity to Meaning: USIM successfully distinguished between high-quality corrections and poor-quality systems that altered meaning to achieve fluency. Scores for the latter dropped significantly (0.19 to 0.39).
  • Feasibility of Automation: While human-annotated graphs provided the most accurate results, the automated TUPA parser achieved a USIM score of 0.70, indicating that reference-less semantic evaluation is viable for rapid system development.
  • Stability of Meaning: The high agreement between human annotators on ungrammatical text suggests that intended meaning is more stable and easier to identify than correct grammatical structure in learner language.

Limitations and Future Work

The primary bottleneck for USIM is the accuracy of the automated UCCA parser, which currently struggles with typos, rare words, and implicit linguistic units. Future work will focus on training parsers specifically on learner language to improve robustness. Additionally, the authors plan to investigate how USIM scores correlate with human perceptions of "conservatism" in editing—the degree to which a corrector stays true to the original intent versus rewriting the sentence.

Summary updated

Abstract

We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality. USim operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USim, by showing that (1) semantic annotation can be consistently applied to ungrammatical text; (2) valid corrections obtain a high USim similarity score to the source; and (3) invalid corrections obtain a lower score.
BibTeX
@inproceedings{choshen2018reference,
  author = {Leshem Choshen and Omri Abend},
  title = {{Reference-less Measure of Faithfulness for Grammatical Error Correction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-2020},
}
Figure