Transfer Fine-tuning for Quality Estimation of Text Simplification

Yuki Hironaka, Tomoyuki Kajiwara, Takashi NinomiyaCOLING 2024

TL;DR

This paper tackles the challenge of training effective quality estimation (QE) models for text simplification, which typically suffer from limited labeled data. They propose a "transfer fine-tuning" approach, pre-training models on a related, large-scale task (identifying sentence simplicity) before fine-tuning on the actual QE task. This method significantly improves QE performance, especially for simplicity, and even shows benefits for fluency and meaning preservation.

What Problem Does This Paper Address?

Text simplification aims to rewrite complex sentences into simpler ones while preserving meaning. Evaluating the quality of these simplified sentences is crucial, but human evaluation is costly and time-consuming. Automatic evaluation metrics like SARI and BLEU often correlate poorly with human judgment and require reference sentences, which are not always available in real-world scenarios. This leads to a need for "reference-less" quality estimation (QE) models.

However, training robust QE models for text simplification is hampered by a lack of large, labeled datasets. Existing datasets like QATS and Simplicity-DA contain only around 600 sentence pairs, which is insufficient to train complex deep learning models effectively. This scarcity of data makes it difficult to achieve high-quality QE, especially for nuanced aspects like fluency and meaning preservation.

Methods

The core of this paper's contribution lies in its transfer fine-tuning strategy. Instead of directly fine-tuning a pre-trained language model (like BERT, RoBERTa, or DeBERTa) on the small QE dataset, they introduce an intermediate step:

  1. Pre-training: The model is first pre-trained on a large-scale, readily available corpus for text simplification. This pre-training task is a pseudo-QE task focused on sentence difficulty estimation. Specifically, the model is trained to classify whether one sentence is simpler than another, using concatenated pairs of complex and simple sentences from monolingual parallel corpora. This task doesn't require explicit QE labels, making it scalable.
  2. Fine-tuning on Relevant Task (Pseudo-QE): The pre-trained model is then fine-tuned on this sentence difficulty estimation task. This step helps the model learn features relevant to simplicity.
  3. Fine-tuning on Target Task (QE): Finally, the model, now adapted to the relevant task, is fine-tuned on the actual, small-scale QE dataset for text simplification. This two-step fine-tuning process aims to leverage the knowledge gained from the large pseudo-QE dataset to improve performance on the target QE task.

The authors use Transformer encoder models and evaluate QE performance using Pearson correlation coefficients for fluency, meaning preservation, and simplicity.

Datasets

  • Pseudo-QE Training:
    • Wikipedia (Wiki-Auto): Used for training.
  • Turk Corpus: Used for validation.
  • Newsela-Auto: Used for both training and validation. These are large-scale, monolingual parallel corpora commonly used for training text simplification models, but they do not contain QE labels.

  • Target QE Evaluation:

    • Simplicity-DA: This is the primary dataset for evaluating the QE models. It consists of 600 sentence pairs, split into 400 for training and 100 each for validation and testing. This dataset is specifically designed for evaluating deep learning-based text simplification models.

Key Insights and Ideas

  • Leveraging Large, Unlabeled Data: The key insight is that a related, large-scale task (sentence difficulty estimation) can be used to pre-train models for a low-resource target task (QE for text simplification), even without explicit QE labels for the pre-training data.
  • "Pseudo-QE" as a Bridge: The sentence difficulty estimation task acts as a valuable intermediate step, teaching the model about sentence complexity and simplicity, which directly benefits the QE-for-simplicity objective.
  • Unexpected Gains: While improved simplicity estimation was expected, the authors were surprised to find that the transfer fine-tuning also led to improvements in fluency and meaning preservation for some models (DeBERTa and BERT). This suggests that learning about sentence structure and complexity can have broader positive effects on overall quality.
  • Data Efficiency: The analysis shows that a significant portion of the benefit from the pseudo-QE task can be achieved with relatively modest amounts of training data (tens of thousands of sentence pairs), making the approach practical for languages with less abundant parallel corpora.

Key Results

  • The proposed transfer fine-tuning method generally improved QE performance on simplicity across different pre-trained models (BERT, RoBERTa, DeBERTa) compared to baseline fine-tuning.
  • For DeBERTa, the method improved fluency and meaning preservation. For BERT, meaning preservation also saw improvements.
  • The analysis of training data size for the pseudo-QE task indicated that peak performance was achieved with 50,000 to 100,000 sentence pairs. Using more data beyond this point did not yield substantial additional gains.
  • The paper also compared their method against older machine learning approaches (Kajiwara-17, Martin-18), showing the advantage of deep learning models with transfer fine-tuning.

Limitations and Future Work

  • Model Dependency: The extent of improvement in fluency and meaning preservation varied across different pre-trained models, suggesting that the effectiveness of transfer fine-tuning might be model-dependent.
  • Future Work:
    • Developing additional training methods specifically targeting fluency and meaning preservation.
  • Extending the approach to quality estimation of text simplification in non-English languages, leveraging the finding that tens of thousands of sentence pairs are sufficient for effective transfer.

Summary updated

BibTeX
@inproceedings{hironaka2024transfer,
  author = {Yuki Hironaka and Tomoyuki Kajiwara and Takashi Ninomiya},
  title = {{Transfer Fine-tuning for Quality Estimation of Text Simplification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure