SIERA: An Evaluation Metric for Text Simplification using the Ranking Model and Data Augmentation by Edit Operations

Hikaru Yamanaka, Kentaro Inui2024 · DOI

TL;DR

This paper introduces SIERA, a novel reference-free automatic evaluation metric for text simplification that uses a ranking model trained on original and simplified sentence pairs. SIERA addresses the high cost and limitations of reference-based metrics by leveraging data augmentation with edit operations to improve its training data, demonstrating superior correlation with human judgments on multiple datasets.

What Problem Does This Paper Address?

Evaluating the quality of text simplification (TS) systems is crucial, but existing automatic evaluation metrics face several challenges. While fluency and meaning preservation are common concerns in text generation, simplicity is a unique and vital aspect of TS. Current metrics for simplicity often require human-written reference texts, which are expensive and time-consuming to create. Furthermore, relying on a single reference can be problematic, as multiple simplified versions of a text can be equally valid. Existing reference-free metrics also struggle to accurately capture simplicity, often showing low correlation with human evaluations. This paper aims to bridge this gap by proposing a new reference-free metric that is both effective and cost-efficient.

Methods

The core of SIERA is a ranking model designed to predict the relative simplicity of sentences. Instead of directly assigning a score, it learns to order pairs of sentences based on their simplicity. This model is trained using parallel corpora of original and simplified sentences. A key innovation is the data augmentation technique that expands the training data. This method utilizes edit operations (token and span level) to generate intermediate sentences that lie in simplicity between the original and the simplified versions. By applying subsets of these edit operations, new sentence pairs are created, effectively increasing the training data size and diversity. The paper also proposes an improvement to the NPRM (Neural Pairwise Ranking Model) architecture, which forms the basis of SIERA, by incorporating both forward and backward sentence pair orders during inference to better utilize learned information.

Datasets

  • Training Data:
    • Newsela: A large dataset of English news articles with manually rewritten simplified versions across four difficulty levels. The authors used pairs from level 1 (original) and level 4 (most simple) for initial training, and also explored using intermediate levels for augmentation.
  • Evaluation Data:
    • Simplicity-DA: A dataset of original sentences from TurkCorpus and simplified sentences generated by six automatic simplification models. Human ratings are continuous (0-100).
  • Human-Likert: Uses TurkCorpus as original sentences, but with manually written simplified sentences. It has 100 sentence pairs with 30 human ratings each.
  • SimpDA2022: Source texts from Wikipedia simplified by two humans and four recent TS models, with three manual ratings per pair.

Key Insights and Ideas

  • Reference-Free Simplicity is Achievable: The paper demonstrates that it's possible to create an effective simplicity metric without relying on human-written references, addressing a significant bottleneck in TS research.
  • Edit Operations as a Bridge: Using edit operations to generate intermediate sentences is a clever way to create more training data that captures the nuanced progression of simplification. This helps the model learn the "in-between" states of simplicity.
  • Quality Over Quantity (Sometimes): While data augmentation is beneficial, the results suggest that the quality of the intermediate sentences (e.g., those derived from human edits) can have a more significant impact than simply increasing the quantity of augmented data.
  • Training-Evaluation Consistency Matters: The performance of SIERA is sensitive to the quality of the simplified sentences in both the training and evaluation datasets. A mismatch in quality can lead to lower correlation with human judgments.
  • Two-Way Inference is Generally Better: Incorporating both forward and backward sentence pair orders during inference generally leads to better performance, suggesting a more robust understanding of simplicity.

Key Results

  • SIERA consistently shows superior correlation with human ratings compared to other metrics (SARI, BLEU, BERTScore, SAMSA, FKGL) on the Human-Likert and SimpDA2022 datasets.
  • On the high-quality subset of Simplicity-DA, SIERA also outperforms other metrics, reinforcing its effectiveness when training and evaluation data quality align.
  • The data augmentation method, particularly with human-written intermediate sentences (+Gold), significantly improves SIERA's performance over a baseline model trained without augmentation.
  • The paper reports mean Pearson and Spearman correlation coefficients for SIERA and other metrics across the three datasets, with SIERA generally achieving higher values, especially on Human-Likert and SimpDA2022. For instance, on Human-Likert, SIERA (+Gold) achieves Pearson correlations of 0.607 and 0.604, significantly higher than many other metrics.

Limitations and Future Work

  • Training Data Quality: The authors acknowledge that SIERA's performance is highly dependent on the quality of the training data. If the training data contains low-quality simplified sentences, the model may struggle to evaluate them accurately. This was observed with the Simplicity-DA dataset, where SIERA performed less well when evaluated against automatically generated sentences of varying quality.
  • Filtering Irrelevant Sentences: The BERTScore-based filtering for intermediate sentences, while helpful, is noted as being limited. The authors suggest that sentence similarity alone might not be sufficient and that future work could explore more effective methods for generating and filtering relevant intermediate sentences, potentially by considering syntactic structure.
  • Domain Shift: The study was primarily conducted on English news articles. The authors suggest that further research is needed to verify SIERA's effectiveness on different domains, as parallel corpora for simplification in various domains are scarce.
  • Dependencies in Edit Operations: The data augmentation method does not fully consider dependencies among edit operations, which can sometimes lead to irrelevant intermediate sentences. Future work could focus on generating more coherent and relevant intermediate sentences by better modeling these dependencies.

Summary updated

BibTeX
@misc{yamanaka2024siera,
  author = {Hikaru Yamanaka and Kentaro Inui},
  title = {{SIERA: An Evaluation Metric for Text Simplification using the Ranking Model and Data Augmentation by Edit Operations}},
  year = {2024},
  doi = {10.63317/5nki5bykwqm7},
}
Figure