GMU at MLSP 2024: Multilingual Lexical Simplification with Transformer Models

Dhiman Goswami, Kai North, Marcos ZampieriNAACL 2024

TL;DR

This paper presents GMU's submission to the MLSP 2024 shared task, tackling multilingual lexical complexity prediction (LCP) and lexical simplification (LS) across 10 languages. They achieved strong results using a cross-lingual weighted ensemble of transformer models for LCP and GPT-4 zero-shot prompting for LS, demonstrating the effectiveness of these approaches in a multilingual setting.

What Problem Does This Paper Address?

The paper addresses the challenge of making text more accessible and understandable across diverse linguistic backgrounds. Lexical Complexity Prediction (LCP) aims to identify words that might be difficult for certain readers, while Lexical Simplification (LS) focuses on replacing these complex words with simpler synonyms without altering the original meaning. This is crucial for inclusive communication and broadening access to information, especially in a multilingual context where language barriers can exacerbate comprehension issues. The MLSP 2024 shared task specifically focuses on these tasks across 10 languages, pushing the boundaries of multilingual NLP for text simplification.

Methods

For Lexical Complexity Prediction (LCP), the authors employed a cross-lingual weighted ensemble of transformer models. This approach leverages the strengths of various pre-trained language models (like mBERT, XLM-R, and language-specific BERT variants) by training them on data from all available languages. The weights for combining these models were determined by the Pearson correlation coefficient, allowing the ensemble to adapt to the nuances of each language. For English LCP, they also experimented with using the CompLex dataset as an additional training resource.

For Lexical Simplification (LS), the primary method was GPT-4 turbo zero-shot prompting. This means they directly asked GPT-4 to simplify text without providing specific examples for training. To further refine the suggestions, they also explored using the top 10 simplifications generated by GPT-4 and BERT-based models. The selection of the best simplification involved using LaBSE sentence embeddings to calculate cosine similarity between the target word and the suggested replacements, ensuring semantic similarity and context preservation.

Datasets

The MLSP shared-task utilized a dataset covering 10 languages: Catalan, English, Filipino, French, German, Italian, Japanese, Portuguese, Sinhala, and Spanish. For both LCP and LS, the data instances consist of a sentence and a specific word from that language, along with either a complexity value (0-1) for LCP or 10 simplified substitutions for LS. The annotators for this dataset came from diverse backgrounds (students, language learners, university faculty, freelancers) and included both native and non-native speakers, aiming for robust annotation. For English LCP, the authors also incorporated the CompLex dataset, which was merged with trial data for training.

Key Insights and Ideas

  • Cross-lingual transfer learning is powerful for LCP: Even with limited language-specific data, an ensemble of models trained across languages can achieve competitive LCP performance. This highlights the ability of transformers to generalize linguistic knowledge.
  • GPT-4's zero-shot capability shines in LS: For lexical simplification, GPT-4's ability to perform the task effectively without explicit fine-tuning on the target data is a significant finding, suggesting its strong understanding of language and simplification principles.
  • Combining models for LS can be tricky: While combining GPT-4 and BERT suggestions for LS seemed promising, the authors found that it sometimes decreased performance. This is attributed to grammatical variations in target words and the difficulty of finding perfect fits from a larger pool of suggestions.
  • Multilingualism is a key challenge: The paper underscores the importance of developing techniques that can generalize across languages, as demonstrated by the MLSP shared task's multilingual nature.

Key Results

  • LCP Performance: GMU's submission achieved ranks from 1st to 5th in LCP across the 10 languages. Specifically, their weighted ensemble approach secured 1st place for English LCP with a Pearson correlation of 0.8497. They also achieved competitive scores for other languages, with Pearson coefficients ranging from 0.1246 (Sinhala) to 0.3494 (All Combined).
  • LS Performance: For LS, GPT-4 zero-shot prompting achieved the best results. They ranked 1st for Sinhala with an Accuracy@1@Top1 score of 0.4182. They also secured 2nd place for German, Spanish, All Combined, Japanese, and Filipino.

Limitations and Future Work

The authors acknowledge that a key limitation is the reliance on cross-lingual transfer due to limited language-specific training data. They suggest that larger, language-specific datasets could further boost performance. Additionally, their error analysis revealed challenges in handling complex word expressions and phrases during LS, indicating that future work should focus on more sophisticated modeling for these cases. They also note that negative R2 scores for some languages indicate that the data struggled to fit the regression model, suggesting potential issues with data distribution or model fit for those specific languages.

Summary updated

BibTeX
@inproceedings{goswami2024gmu,
  author = {Dhiman Goswami and Kai North and Marcos Zampieri},
  title = {{GMU at MLSP 2024: Multilingual Lexical Simplification with Transformer Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure