Reference-less Measure of Faithfulness for Grammatical Error Correction

Leshem Choshen, Omri Abend

Introduction

Evaluating Grammatical Error Correction (GEC) systems is a complex task. Traditionally, researchers rely on Reference-Based Measures (RBMs), which compare system outputs to human-curated gold standards. However, language is flexible; a single error can often be corrected in multiple valid ways. RBMs are limited by the coverage of their reference sets, often penalizing correct but unreferenced outputs.

To address this, Reference-Less Measures (RLMs) have emerged. While existing RLMs effectively evaluate the grammaticality of an output, they often overlook faithfulness—the preservation of the original meaning. A system might produce a perfectly fluent sentence that completely alters the user's intent. This paper introduces USIM, a novel reference-less metric designed to measure semantic faithfulness in GEC.

The USIM Metric

USIM evaluates faithfulness by comparing the semantic structures of the source sentence and the corrected output. It leverages the Universal Conceptual Cognitive Annotation (UCCA) framework. Unlike syntactic trees, which can be highly irregular in "Learner Language" (text produced by non-native speakers), UCCA focuses on semantic units such as scenes, participants, and processes.

The USIM score is calculated through a three-step process:

  1. Leaf Alignment: Tokens (leaves of the UCCA graph) are aligned using edit distance, accounting for the fact that most GEC edits are minor.
  2. Node Alignment: Higher-level semantic nodes are aligned based on the overlap of their underlying tokens.
  3. F-score Calculation: A Directed Acyclic Graph (DAG) F-score is computed based on the matching edges between the two semantic representations.

As shown in the figure above, USIM abstracts away from surface-level syntactic changes (like the shift from a prepositional phrase to a double object construction) to focus on whether the core participants and processes remain the same.

Experimental Results

The authors validated USIM using both human annotations and automated parsing via the TUPA parser. Key findings include:

  • Robustness to Errors: Human annotators achieved a high Inter-Annotator Agreement (IAA) DAG F-score of 0.845 on ungrammatical learner text, proving that semantic meaning can be reliably extracted even from broken English.
  • Sensitivity to Faithfulness: Valid human corrections typically yielded USIM scores between 0.84 and 0.92. In contrast, when tested on poor GEC systems that hallucinated or deleted information, the automated USIM scores dropped significantly (as low as 0.19), effectively penalizing unfaithful edits.
  • Viability of Automated Parsing: Even using an off-the-shelf parser not specifically trained on learner language, automated USIM achieved a score of ~0.70 on valid corrections, mirroring the parser's baseline accuracy and demonstrating its immediate utility as an evaluation tool.

Conclusion

USIM provides a critical missing piece in GEC evaluation by focusing on semantic faithfulness without requiring expensive human references. By utilizing the UCCA framework, it offers a robust way to ensure that "corrections" do not inadvertently change the speaker's meaning. Future work will involve refining automated parsers for learner language and exploring how humans balance the trade-offs between fluency and faithfulness.

Summary updated

BibTeX
@inproceedings{choshen2018reference,
  author = {Leshem Choshen and Omri Abend},
  title = {{Reference-less Measure of Faithfulness for Grammatical Error Correction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-2020},
}
Figure