Label Confidence Weighted Learning for Target-level Sentence Simplification

Xin Ying Qiu, Jingshen ZhangEMNLP 2024 · DOI

Introduction

Sentence simplification is not a monolithic task. Different audiences—children, non-native speakers, or individuals with cognitive disabilities—require different levels of linguistic complexity. Target-level (or multi-level) sentence simplification addresses this by generating text tailored to specific proficiency levels. However, progress in this field is often hindered by the scarcity of high-quality, level-annotated datasets like Newsela-auto.

The Problem of Noisy Labels

To overcome data scarcity, researchers frequently employ "pseudo-labeling," where a classifier labels large, unannotated paraphrase datasets such as ParaNMT. Unfortunately, this introduces "label noise." If a classifier misidentifies a sentence's level, the subsequent generation model learns from these errors, significantly degrading performance. This research introduces a framework to utilize massive, noisy data without allowing these errors to compromise the final model's quality.

Methodology: Label Confidence Weighted Learning (LCWL)

The authors propose Label Confidence Weighted Learning (LCWL), a three-stage framework designed to leverage large-scale noisy data effectively by incorporating a confidence-weighting scheme directly into the training loss.

The LCWL framework: (1) Training a multi-level classifier, (2) labeling a large paraphrase dataset, and (3) confidence-weighted training of an encoder-decoder model.
  1. Multi-level Classifier Training: A BERT-based classifier is first trained on the high-quality Newsela-auto dataset to predict simplification levels.
  2. Pseudo-Labeling: This classifier is applied to 1 million sentence pairs from ParaNMT. For each pair, the predicted level and the classifier's confidence score are recorded.
  3. Confidence-Weighted Training: Instead of treating pseudo-labels as absolute ground truth, LCWL weights the training loss of a BART-based encoder-decoder model. The weight is the geometric mean of the classifier’s historical precision for a given level and the specific sentence's confidence score. This ensures the model prioritizes high-confidence, historically accurate labels over noisier data points.

Key Results

The LCWL approach was evaluated against state-of-the-art baselines using metrics like SARI (simplicity and meaning preservation) and LENS (a modern, learnable metric for simplification quality).

Comprehensive evaluation results across four simplification levels (Simp-1 to Simp-4). LCWL variants consistently achieve top ranks in both unsupervised and supervised settings.
  • Unsupervised Lead: In purely unsupervised settings (where Newsela is not used for generator training), LCWL achieved the top rank across almost all evaluation metrics compared to established baselines like MUSS and FUDGE.
  • Competitive with Large Language Models: When fine-tuned (LCWL+FT), the model outperformed few-shot GPT-3.5-Turbo in several key metrics, particularly in its ability to estimate and achieve the correct simplicity level (measured by $\Delta$SLE).
  • Synergy with SCE: Combining LCWL with Symmetric Cross Entropy (SCE) yielded the best overall average rank, proving that noise mitigation at both the classification and generation stages is highly effective. Qualitative analysis also showed that LCWL-trained models excel at sentence splitting, a hallmark of high-quality human simplification.

Limitations and Future Work

While effective, the pipeline depends heavily on the initial classifier's accuracy. Additionally, the study was limited to English due to the availability of paraphrase datasets and pre-trained classifiers. Future work may involve applying LCWL to other data-scarce generation tasks such as style transfer or specialized summarization in low-resource languages.

Summary updated

BibTeX
@inproceedings{qiu2024label,
  author = {Xin Ying Qiu and Jingshen Zhang},
  title = {{Label Confidence Weighted Learning for Target-level Sentence Simplification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.999},
}
Figure