ALEXSIS-PT: A New Resource for Portuguese Lexical Simplification

Kai North, Marcos Zampieri, Tharindu RanasingheCOLING 2022

TL;DR

This paper introduces ALEXSIS-PT, a novel, large-scale dataset for Portuguese lexical simplification, featuring newspaper articles and a high number of candidate substitutions per complex word. They also evaluate several state-of-the-art transformer models, finding that BERTimbau, a Portuguese-specific model, outperforms multilingual models, suggesting the importance of language-specific pre-training for this task.

What Problem Does This Paper Address?

Lexical simplification (LS) aims to make texts more accessible by replacing complex words with simpler alternatives. This is crucial for individuals with low literacy, learning disabilities, or those learning a new language. However, developing effective LS systems requires high-quality datasets for training and evaluation. Existing Portuguese LS datasets either lack candidate substitutions for complex words, are domain-specific (e.g., children's books), or offer a limited number of substitutions, hindering the evaluation of modern, powerful language models. This paper addresses this gap by introducing ALEXSIS-PT, a comprehensive dataset designed to facilitate research in Portuguese lexical simplification.

Methods

The authors introduce ALEXSIS-PT, a new dataset compiled following the ALEXSIS protocol. This involved:

  • Word Alignment: Manually aligning original and simplified sentences from the PorSimplesSent dataset to identify complex words.
  • Crowdsourcing: Using Amazon MTurk annotators in Brazil to generate candidate substitutions for these identified complex words.
  • Linguistic Filtering: A linguist reviewed the generated substitutions, removing those that were identical to the complex word, not in Portuguese, or inappropriate.

For evaluating substitute generation (SG), they employed a Masked Language Modeling (MLM) strategy, similar to previous work. They masked the complex word in a sentence and fed both the original and masked sentences to transformer models to predict the masked token.

Datasets

  • ALEXSIS-PT: The primary contribution, a new multi-candidate dataset for Brazilian Portuguese lexical simplification. It contains 9,605 candidate substitutions for 387 complex words extracted from newspaper articles. This dataset is notable for its high average number of candidate substitutions per complex word (25).
  • SIMPLEX-PB 3.0: An existing Portuguese LS dataset used for comparison. It features substitutions for over 1,500 complex words but has a limited number of ranked substitutions (5 for most words) and is domain-specific (children's books).
  • ALEXSIS-ES: The Spanish counterpart to ALEXSIS-PT, used for cross-lingual model considerations.
  • Simple Wikipedia: Mentioned as a common resource for English LS, used for training unsupervised models.
  • Newsela Corpus: Used in conjunction with neural networks for English LS.
  • PorSimples Project: The foundational work for Portuguese LS, which created an initial dataset but lacked candidate substitutions.
  • PorSimplesSent: A dataset derived from the PorSimples project, used as the source for ALEXSIS-PT.

Key Insights and Ideas

  • Language-Specific Models Shine: BERTimbau, a model pre-trained specifically on Brazilian Portuguese data, significantly outperforms multilingual models (mDistilBERT, mBERT, XLM-R) on the ALEXSIS-PT dataset. This suggests that for tasks like lexical simplification, language-specific pre-training can be more effective than general multilingual models, especially when dealing with language-specific nuances.
  • Dataset Characteristics Matter: Models performed better on ALEXSIS-PT than on SIMPLEX-PB 3.0. This is attributed to ALEXSIS-PT's newspaper genre (more general) and its larger number of candidate substitutions, which provides a richer evaluation environment compared to the domain-specific and limited SIMPLEX-PB 3.0.
  • Lemmatization Aids SG: Reducing words to their lemmas before substitute selection and ranking improved model performance. This highlights the impact of Portuguese morphology and suggests that future SG systems should incorporate lemmatization to handle inflectional variations more effectively.
  • More Candidates, Better Recall: As the number of candidate substitutions generated increases (from top-k=1 to top-k=100), models show higher potential and recall scores. This is intuitive, as a larger pool of candidates increases the likelihood of including correct ground truth labels.

Key Results

  • BERTimbau Dominance: BERTimbau achieved the highest performance across all evaluation metrics for substitute generation on ALEXSIS-PT.
  • ALEXSIS-PT vs. SIMPLEX-PB 3.0: Models generally showed better performance on ALEXSIS-PT compared to SIMPLEX-PB 3.0. For instance, BERTimbau achieved a precision of 0.212 for top-k=3 on ALEXSIS-PT.
  • Lemmatization Impact: Lemmatization led to performance increases across all metrics. BERTimbau's F1-score improved by 0.002 for top-k=3 substitutions after lemmatization.
  • AWFR: BERTimbau achieved the highest Average Weighted Frequency Rank (AWFR) of 0.185 after lemmatization, indicating its predicted substitutions were most aligned with annotator frequency rankings.

Limitations and Future Work

  • Limitations:
    • The study focused on substitute generation (SG) and did not implement a full LS pipeline (including substitute selection and ranking).
  • Multilingual models sometimes produced substitutions in European Portuguese or other languages entirely, indicating a need for better language control.
  • Future Work:
    • Implementing a complete LS pipeline on ALEXSIS-PT, incorporating substitute selection and ranking.
  • Exploring transfer learning and developing multilingual LS systems, potentially leveraging the ALEXSIS-ES dataset.
  • Further investigating the impact of morphology and developing methods to produce contextually appropriate lexical simplifications.

Summary updated

BibTeX
@inproceedings{north2022alexsis,
  author = {Kai North and Marcos Zampieri and Tharindu Ranasinghe},
  title = {{ALEXSIS-PT: A New Resource for Portuguese Lexical Simplification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure