ISEP_Presidency_University at MLSP 2024 Shared Task: Using GPT-3.5 to Generate Substitutes for Lexical Simplification

Benjamin Dutilleul, Mathis Debaillon, Sandeep MathiasNAACL 2024

TL;DR

This paper presents a system for lexical simplification that leverages GPT-3.5 to generate contextually appropriate synonyms for complex words. The system achieved top results in Catalan and Portuguese and runner-up in English, French, and Italian in the MLSP 2024 Shared Task, demonstrating the effectiveness of large language models for this educational and accessibility-focused task.

What Problem Does This Paper Address?

The paper tackles the challenge of lexical simplification, a crucial component of text simplification aimed at making written content more accessible. This is particularly important for diverse audiences, including young learners, individuals with language disabilities, and second-language learners. Lexical simplification involves replacing difficult words with simpler, contextually relevant synonyms. While syntactic simplification focuses on sentence structure, this work hones in on the word level. The authors participated in the Multilingual Lexical Simplification Pipeline (MLSP) Shared Task, which involved two main problems: Lexical Complexity Prediction (identifying complex words) and Lexical Simplification (generating ranked substitutes). This paper specifically focuses on the latter.

Methods

The core of the system relies on GPT-3.5, a powerful large language model. Due to the immense size of GPT-3.5, fine-tuning is impractical, so the authors employed prompt engineering. They developed a system that first detects the language of the input text and then selects an appropriate prompt template. Three types of prompts were explored:

  • Context-Free Prompt: Asks for synonyms of a word without any surrounding text, testing general synonym knowledge.
  • Zero-Shot Prompt: Provides the context and the complex word, asking for simpler alternatives without any examples. This gauges the model's ability to infer simplification from context alone.
  • Single-Shot/Few-Shot Prompt: Includes one or more examples of complex words and their simplified synonyms to guide the model towards the desired output format and style.

After generating potential substitutes using these prompts, the system cleans the predictions, aggregates them from different prompts, and ranks them to produce the final output of up to 10 synonyms.

Datasets

The authors utilized trial data provided for the MLSP 2024 Shared Task, consisting of a small number of context-complex word pairs for each language. Their systems were then evaluated on test sets of varying sizes for six languages: English, Catalan, French, German, Italian, and Portuguese. Table 1 in the paper details the sizes of these testing datasets.

Key Insights and Ideas

  • GPT-3.5 as a powerful tool for lexical simplification: The paper strongly suggests that large language models, when prompted effectively, can achieve state-of-the-art results in lexical simplification across multiple languages.
  • Prompt engineering is key for LLMs: The success of the system hinges on carefully crafted prompts that guide GPT-3.5 to generate relevant and simplified synonyms. The choice of prompt type (context-free, zero-shot, few-shot) influences the model's output.
  • Multilingual capabilities are achievable: The system demonstrated strong performance across several languages, highlighting the potential for LLMs to handle linguistic diversity in simplification tasks.
  • Challenges in multilingual prompt creation: Translating prompts from English to other languages using tools like Google Translate can introduce complexities and require careful handling.

Key Results

The system achieved best performances in Catalan and Portuguese and was a runner-up in English, French, and Italian in the MLSP 2024 Shared Task. The paper reports these results in Table 2, comparing their system's performance (highlighted in blue and referred to as "ISEP_PU") against other participating systems based on an aggregate metric derived from MAP@K, Accuracy@k@top1, and Potential@k. While strong across most languages, performance was notably lower in German.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Computational Intensity: LLMs like GPT-3.5 are computationally expensive, requiring significant GPU power for operation.
  • User-Independent Simplification: Current LLM outputs are not personalized to specific user needs (e.g., young learners vs. second-language learners). Future work could involve instruction tuning to tailor simplification based on user profiles or cognitive information.
  • Orthographic Challenges: The system's participation was limited to languages using the Roman script. Future work could explore extending the system to languages with different orthographies (e.g., Bengali, Japanese).
  • Improving Prompting: Further research into prompt engineering, particularly for few-shot examples, could lead to more nuanced and user-specific simplifications.

Summary updated

BibTeX
@inproceedings{dutilleul2024isep,
  author = {Benjamin Dutilleul and Mathis Debaillon and Sandeep Mathias},
  title = {{ISEP\_Presidency\_University at MLSP 2024 Shared Task: Using GPT-3.5 to Generate Substitutes for Lexical Simplification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure