CEScore: Simple and Efficient Confidence Estimation Model for Evaluating Split and Rephrase

AlMotasem Bellah Al Ajlouni, Jinlong Li

Abstract

The split and rephrase (SR) task aims to divide a long, complex sentence into a set of shorter, simpler sentences that convey the same meaning. This challenging problem in NLP has gained increased attention recently because of its benefits as a pre-processing step in other NLP tasks. Evaluating the quality of SR is challenging, as there is no automatic metric fit to evaluate this task. In this work, we introduce CEScore, a novel statistical model to automatically evaluate the SR task. By mimicking the way humans evaluate SR, CEScore provides four metrics ($S_{score}$, $G_{score}$, $M_{score}$, and $CE_{score}$) to assess simplicity, grammaticality, meaning preservation, and overall quality, respectively. In experiments with 26 models, CEScore correlates strongly with human evaluations, achieving 0.98 in Spearman correlations at the model level. This underscores the potential of CEScore as a simple and effective metric for assessing the overall quality of SR models.

Introduction

The "Split and Rephrase" (SR) task is a critical component of text simplification, involving the decomposition of complex sentences into multiple shorter, more accessible ones without altering the core message. Despite its utility in downstream tasks like machine translation and information extraction, SR lacks a robust evaluation framework.

Traditional evaluation relies on two flawed approaches. Reference-based metrics, such as BLEU or BERTscore, require expensive human-written gold standards and often penalize the structural changes inherent to simplification. Conversely, manual human evaluation, while accurate, is prohibitively slow and expensive. To bridge this gap, we propose CEScore, a reference-less "Confidence Estimation" metric that evaluates SR quality by comparing the model output directly against the original complex input.

Methodology

CEScore is designed to emulate human intuition across three fundamental dimensions: simplicity, grammaticality, and meaning preservation. The final quality score, $CE_{score}$, is calculated as the geometric mean of these sub-metrics.

Algorithm 1: The high-level CEScore model architecture combining simplicity, grammaticality, and meaning preservation.

Simplicity ($S_{score}$)

Simplicity is measured via the Text Simplicity Score (TSS), which integrates two components:

  1. Sentence Length Score (SLS): A linear penalty is applied to sentences falling outside the human-perceived "sweet spot" of 5 to 25 words.
  2. Average Sentence Familiarity (ASF): This utilizes Zipf-values from the SUBTLEX-US database to ensure the vocabulary remains accessible to a general audience.

Meaning Preservation ($M_{score}$)

Unlike simple word-overlap metrics, $M_{score}$ employs an entropy-based weighting system. It prioritizes information-dense words (e.g., proper nouns) over common stop words. If a model omits a rare, high-information word, the $M_{score}$ is significantly reduced.

Algorithm 5: The MScore function, which calculates meaning preservation using weighted sums based on word information content.

Grammaticality ($G_{score}$)

To avoid the computational overhead of deep semantic parsing, CEScore uses a SemiMatch algorithm. It assumes the source sentence is grammatically sound and looks for n-gram overlaps (sequences of 4 to 7 words) between the output and the source. This rewards partial matches, allowing for the reordering and substitution necessary for effective rephrasing.

Evaluation and Results

We validated CEScore using the Sulem et al. (2018) benchmark, which contains human ratings for 1,820 sentences generated by 26 different models. Our results demonstrate that CEScore significantly outperforms existing metrics:

  • High Correlation: At the model level, CEScore achieved a Spearman correlation of 0.98 and a Pearson correlation of 0.95 with human judgments.
  • Simplicity Accuracy: While traditional metrics like SARI and BLEU often show negative correlations with human simplicity ratings, CEScore maintained a strong positive correlation (0.83–0.86).
  • Efficiency: CEScore proved superior to SAMSA, the previous state-of-the-art reference-less metric, by avoiding the need for slow semantic parsers while providing more accurate assessments.

Conclusion and Future Work

CEScore offers a lightweight, highly accurate, and reference-less alternative for evaluating Split and Rephrase tasks. By focusing on the transformation from the source sentence rather than comparison to a static reference, it captures the nuances of simplification more effectively than previous methods. Future research will explore the integration of contextual embeddings from Large Language Models (LLMs) to further refine lexical sensitivity and synonym detection.

Summary updated

Abstract

The split and rephrase (SR) task aims to divide a long, complex sentence into a set of shorter, simpler sentences that convey the same meaning. This challenging problem in NLP has gained increased attention recently because of its benefits as a pre-processing step in other NLP tasks. Evaluating quality of SR is challenging, as there no automatic metric fit to evaluate this task. In this work, we introduce CEScore, as novel statistical model to automatically evaluate SR task. By mimicking the way humans evaluate SR, CEScore provides 4 metrics (Sscore, Gscore, Mscore, and CEscore) to assess simplicity, grammaticality, meaning preservation, and overall quality, respectively. In experiments with 26 models, CEScore correlates strongly with human evaluations, achieving 0.98 in Spearman correlations at model-level. This underscores the potential of CEScore as a simple and effective metric for assessing the overall quality of SR models.
BibTeX
@misc{ajlouni2023cescore,
  author = {AlMotasem Bellah Al Ajlouni and Jinlong Li},
  title = {{CEScore: Simple and Efficient Confidence Estimation Model for Evaluating Split and Rephrase}},
  year = {2023},
  eprint = {2312.01356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.01356},
}
Figure