ANU at MLSP-2024: Prompt-based Lexical Simplification for English and Sinhala

Sandaru Seneviratne, Hanna SuominenNAACL 2024

TL;DR

This paper explores using prompt engineering with large generative AI models (like GPT-3.5) for lexical simplification, a task focused on replacing complex words with simpler ones while preserving meaning and syntax. The authors found that prompt-based methods can be effective, particularly for lexical substitution, and demonstrated comparable performance across English and Sinhala, highlighting the potential of these models for improving text accessibility.

What Problem Does This Paper Address?

Making complex text understandable is crucial for accessibility, especially for individuals with limited language proficiency or literacy skills. Lexical simplification aims to achieve this by replacing difficult words with easier alternatives without altering the sentence structure. This task typically involves two sub-tasks: identifying complex words and then substituting them. While traditional methods have evolved from rule-based systems to sophisticated deep learning models, the advent of powerful generative AI models has opened new avenues. This paper investigates how prompt engineering, a technique that guides these large language models (LLMs) with specific instructions, can be leveraged for both identifying complex words and generating simpler substitutions, particularly for English and Sinhala.

Methods

The core of this research lies in prompt engineering applied to a generative AI model (GPT-3.5-turbo-instruct). The authors explored three prompt-based methodologies for both lexical complexity prediction and lexical substitution:

  • Zero-shot: The model receives only the input text and the target word/sentence, with no examples.
  • One-shot: The model receives the input text and one example of a similar task.
  • Few-shot: The model receives the input text and three examples of similar tasks.

For lexical complexity prediction, prompts were designed to elicit a numerical complexity score (0-1) for target words within their context. For lexical substitution, prompts asked the model to generate a list of simpler alternative words for a given target word in its context. An ensemble approach, combining results from multiple prompt templates, was also explored for substitution.

Datasets

The experiments utilized datasets from the MLSP-2024 shared task for both English and Sinhala.

  • Lexical Complexity Prediction: Trial subsets of 30 sentences each were used for prompt template creation (one-shot and few-shot). Test subsets of 600 sentences each were used for evaluation.
  • Lexical Substitution: Trial subsets of 30 samples each were used for prompt template creation. Test subsets of 570 samples for English and 600 samples for Sinhala were used for evaluation.

Key Insights and Ideas

  • Prompt Engineering as a Viable Alternative: The study demonstrates that prompt engineering with LLMs can offer a straightforward and effective approach to lexical simplification, potentially simplifying the development pipeline compared to training complex custom models.
  • Cross-Lingual Comparability: The approach showed comparable performance across English and Sinhala, suggesting that prompt-based methods can be adapted to languages with varying linguistic resources and complexities.
  • Substitution Strengths: The research highlights particular strengths in lexical substitution, where prompt-based methods, especially few-shot and ensemble approaches, yielded promising results.
  • Context is Key (But How Much?): The performance varied across zero-shot, one-shot, and few-shot approaches, indicating that the amount and quality of contextual information provided in the prompt significantly influence the model's predictions.
  • Sinhala's Unique Challenges: The performance differences between English and Sinhala suggest that factors like data availability and language complexity play a critical role. For Sinhala, the one-shot approach seemed more beneficial, possibly due to its ability to leverage limited additional context effectively.

Key Results

  • Lexical Complexity Prediction: The prompt-based methods did not outperform top submissions in the MLSP-2024 shared task. The best prompt-based approach for English achieved a Pearson's R of 0.3358, while for Sinhala, the one-shot approach yielded a Pearson's R of 0.0534.
  • Lexical Substitution (English): The few-shot approach demonstrated strong performance, achieving results comparable to the best submissions. The Accuracy@1 score was 0.5105.
  • Lexical Substitution (Sinhala): The ensemble approach for Sinhala did not achieve satisfactory performance, indicating potential challenges in capturing the language's specific lexical patterns.

Limitations and Future Work

The authors acknowledge several limitations:

  • Resource Requirements: The reliance on GPT-based models, accessed via API, incurred costs and raised ethical concerns regarding data privacy and transparency.
  • Prompt Sensitivity: Results were sensitive to the prompt template, the examples included, and the parameters used, necessitating further investigation into optimal prompt design.
  • Performance Gap: The prompt-based methods for lexical complexity prediction did not reach the performance levels of specialized models.

Future work could focus on:

  • Refining Prompt Tuning: Developing more sophisticated prompt tuning methods to enhance effectiveness.
  • Investigating Influencing Factors: Further exploring the impact of data availability, language complexity, and sample quality on simplification outcomes.
  • Improving Sinhala Performance: Addressing the subpar performance for Sinhala lexical substitution, potentially through more tailored prompt strategies or data augmentation.

Summary updated

BibTeX
@inproceedings{seneviratne2024anu,
  author = {Sandaru Seneviratne and Hanna Suominen},
  title = {{ANU at MLSP-2024: Prompt-based Lexical Simplification for English and Sinhala}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure