RALS: Resources and Baselines for Romanian Automatic Lexical Simplification

Fabian Anghel, Petru Theodor Cristea, Claudiu Creangă, Sergiu NisioiEMNLP 2025

TL;DR

This paper introduces the first comprehensive dataset and benchmarks for Romanian lexical complexity prediction (LCP) and lexical simplification (LS). They developed novel datasets and a hybrid rule-based system that outperforms LLM-based approaches for Romanian, highlighting the challenges and opportunities in low-resource language NLP.

What Problem Does This Paper Address?

Text simplification, particularly lexical simplification (LS), aims to make text easier to understand by replacing complex words with simpler alternatives. While LS has seen significant progress in well-resourced languages, it remains largely underexplored for low-resource languages like Romanian. This gap hinders effective communication and information access for Romanian speakers with diverse literacy levels. Existing approaches often rely on large language models (LLMs), which can be costly, lack transparency, and struggle with Romanian's specific linguistic nuances. This paper addresses this by creating crucial resources and developing a robust LS system tailored for Romanian.

Methods

The authors tackle the problem through a multi-pronged approach:

  • Data Collection: They created three Romanian datasets for lexical complexity prediction:
  • HT (Human Translation): Direct Romanian translations of English LCP samples, ensuring semantic equivalence.
  • WT (Word-level Translation): Samples from a Romanian corpus, aiming to assess the impact of translation on complexity.
  • RoLCP (Romanian Lexical Complexity Prediction): New samples from diverse Romanian texts, specifically annotated for complexity. They also developed a dataset for lexical simplification candidates.

  • Complexity Annotation: Annotators rated word complexity on a 5-point scale, which was converted to numerical values. They used a pairwise ranking approach to order simplification suggestions from simple to complex.

  • Hybrid Simplification System (DexFlex): This system combines several components:
  • Synonym Suggestion: Utilizes the dexonline dictionary and contextual word embeddings (from Romanian BERT) to find relevant synonyms.
  • Word-Sense Disambiguation: Compares sentence context with cached embeddings of alternative meanings to select the most appropriate synonym.
  • Inflection: Inflects suggested synonyms to match the grammatical context of the sentence, leveraging spaCy and pre-defined rules.
  • Ranking: The LCP pipeline is used to rank the candidate synonyms by simplicity.

Datasets

The paper introduces and utilizes the following datasets:

  • English LCP Data: Sourced from Wikibooks, used as a baseline for translation.
  • MultiLS Dataset: A multilingual LCP dataset (Shardlow et al., 2024c) that served as a foundation for the Romanian datasets.
  • Representative Corpus of Romanian: Used to extract sentences for the WT and RoLCP datasets.
  • New Romanian Datasets:
    • HT: 569 word-sentence pairs.
  • WT: 1,765 word-sentence pairs.
  • RoLCP: 1,587 word-sentence pairs.
  • ROLS: A dataset for lexical simplification candidates.

In total, 3,921 annotated samples for LCP on Romanian were created.

Key Insights and Ideas

  • Translation Shifts Complexity: Direct translation from English to Romanian can alter perceived lexical complexity. Words that are medium complex in English might be perceived as simpler in Romanian, suggesting that cross-lingual transfer of complexity scores is not straightforward.
  • Hybrid Systems Excel for Low-Resource Languages: A hybrid rule-based system (DexFlex) that combines dictionaries, contextual embeddings, and grammatical rules outperforms large language models (LLMs) like GPT-4 and fine-tuned BERT for Romanian lexical simplification. This suggests that for low-resource languages, carefully engineered systems can be more effective than relying solely on massive pre-trained models.
  • Handcrafted Features Remain Relevant for LCP: Lexical complexity can be reliably predicted using handcrafted features, comparable to LLM-based models. This advocates for simpler baseline models and the integration of linguistic resources like dictionaries.
  • Challenges of LLM Prompting: Prompting LLMs for candidate suggestions can lead to issues with correct inflection and may result in hallucinations, making them less reliable for Romanian LS.

Key Results

  • LCP Performance: A simple Ridge regressor baseline with handcrafted features achieved a Pearson correlation of 0.71 on the RoLCP dataset for lexical complexity prediction. Cross-dataset evaluation showed moderate generalization, with WT → RoLCP and RoLCP → WT yielding the best transfer.
  • LS Performance: The DexFlex framework consistently outperformed other approaches across various metrics (MAP@N, Potential@N, ACC@N@top_gold_1) in lexical simplification tasks. For instance, DexFlex achieved a MAP@1 of 0.41, significantly higher than Apertus-8B (0.21) and RoLlama-8B (0.4). While RoLlama-8B showed slightly better accuracy on top-gold metrics, DexFlex demonstrated superior overall ranking and coverage.
  • Romanian Complexity Distribution: The Romanian datasets showed a larger proportion of words annotated as simple compared to English, with a moderately negative correlation between Zipf frequency and complexity.

Limitations and Future Work

  • Dataset Differences: The three Romanian LCP datasets (HT, WT, RoLCP) have some differences in annotation methodology and size, which could impact model generalization.
  • Annotator Demographics: The annotators were educated young adults, which might limit the generalizability of models trained on this data to broader real-world applications.
  • DexFlex Limitations: DexFlex faces challenges in handling words with multiple parts of speech that share the same form, though it attempts to mitigate this using spaCy. Distinguishing nuances between similar words remains difficult.
  • Future Work: The authors advocate for the development of simpler baseline models and the integration of dictionaries into NLP pipelines. They also suggest further research into hybrid approaches and exploring more robust methods for Romanian text simplification.

Summary updated

BibTeX
@inproceedings{anghel2025rals,
  author = {Fabian Anghel and Petru Theodor Cristea and Claudiu Creangă and Sergiu Nisioi},
  title = {{RALS: Resources and Baselines for Romanian Automatic Lexical Simplification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure