TL;DR
This paper details the RETUYT-INCO team's participation in the BEA-MLSP 2024 shared task on lexical simplification and complexity prediction. They explored various approaches, from traditional embeddings to fine-tuning large language models (LLMs) like Mistral-7B, demonstrating that LLMs, particularly when fine-tuned on synthetic or existing data, achieve state-of-the-art results despite significant data scarcity.
What Problem Does This Paper Address?
The paper tackles the challenge of lexical simplification, which aims to replace complex words in text with simpler alternatives to improve readability and accessibility. This is crucial for language learners, children, and individuals with reading difficulties. The specific problem addressed is twofold: predicting the complexity of a word within its context (Task 1) and finding suitable simpler replacements (Task 2). A major hurdle in this field is the scarcity of high-quality, multilingual data, especially for training and evaluation. The BEA-MLSP 2024 shared task provided a limited trial dataset (only 30 examples per language) and no training data, forcing participants to rely on external resources.
Methods
The RETUYT-INCO team experimented with a range of methods:
- Word Embeddings and Frequency Baselines: They started with traditional approaches using static word embeddings (Polyglot, SBW-vectors, googlenewsvectors) and word frequency lists. For Task 1, complexity was inferred from the frequency of nearby words. For Task 2, they found simpler words from the embedding space and sorted them by frequency or distance.
- Contextual Embeddings: They explored using pre-trained contextual embeddings like BERT and ALBERT. These models were used to generate masked token predictions, which were then ranked by probability or word frequency.
- Fine-tuning Large Language Models (LLMs): This was their primary focus.
- Mistral-7B Fine-tuning: They fine-tuned the Mistral-7B Instruct v0.2 model using Low-Rank Adaptation (LoRA). This was done on:
- Synthetic Data: Generated using Claude 3 Opus, creating a dataset of complexity scores and simpler alternatives for Spanish, English, and Portuguese. This synthetic data generation involved a multi-step process of generating sentences, selecting target words, and then prompting Claude 3 for analysis, scores, and alternatives, incorporating Chain-of-Thought (CoT) reasoning.
- Existing Datasets: They also fine-tuned on the LCP2021 dataset for English and the EASIER_CORPUS for Spanish (though this latter dataset required significant adaptation due to its binary complexity labels).
- BERT and MLP Models: For Task 1, they used BERT (English) and BETO (Spanish) embeddings as input to Multilayer Perceptron (MLP) models.
- Groq Platform: As a final experiment for Task 2 in Spanish, they leveraged the Groq platform to prompt LLMs like LLAMA, GEMMA, and MIXTRAL, using a one-shot mechanism and then ranking the generated alternatives by CREA corpus word frequencies.
Datasets
- Trial Data: A small dataset of 30 examples per language from the BEA-MLSP 2024 shared task was used as a development set.
- Synthetic Dataset: A new dataset was generated using Claude 3 Opus for Spanish, English, and Portuguese, containing complexity scores and simpler alternatives. This dataset comprised 2211 examples.
- Existing Datasets:
- LCP2021 (Lexical Complexity Prediction): Used for fine-tuning English models.
- EASIER_CORPUS: Used for Spanish, though its binary complexity labels required adaptation.
- Spanish Billion Word Corpus, googlenewsvectors, ConLL17 corpus: Used for word embeddings.
- Wiktionary Spanish frequency list, Kaggle English Word Frequency dataset, wordfreq library, CREA corpus: Used for word frequency data.
- MultiLS Framework: The underlying framework for the shared task dataset annotation.
Key Insights and Ideas
- LLMs Excel in Data Scarcity: The most significant finding is that fine-tuning LLMs like Mistral-7B is highly effective, even with limited data, outperforming traditional methods.
- Synthetic Data Generation is Viable: The paper demonstrates that LLMs can be used to generate high-quality synthetic data for lexical simplification tasks, effectively augmenting scarce real-world data.
- Context is King: Contextual embeddings and LLMs that inherently process context significantly outperform non-contextual methods.
- Chain-of-Thought (CoT) and Self-Consistency (SC) Impact: While CoT was explored, the results suggested that SC (Self-Consistency) techniques, particularly for inference, were more beneficial for Mistral-7B fine-tuning.
- Multilingualism is Challenging but Achievable: The task covered 10 languages, and the team's success with Mistral-7B fine-tuned on multilingual synthetic data highlights the potential for robust multilingual NLP systems.
Key Results
- Task 1 (Complexity Prediction): The best results were achieved by fine-tuning Mistral-7B on the LCP2021 English dataset, yielding a Pearson correlation of 0.8061 and Spearman of 0.7596. Other Mistral-7B variants also performed well.
- Task 2 (Lexical Simplification): The top-performing models for Task 2 were also based on fine-tuning Mistral-7B. For Spanish, a model fine-tuned on a synthetic en-es dataset achieved a MAP@1 of 0.9333. For English, a similar model achieved a MAP@1 of 0.5947.
- Comparison: The fine-tuned Mistral-7B models consistently outperformed baseline methods using static embeddings or BERT/ALBERT embeddings.
Limitations and Future Work
- Time Constraints: The authors acknowledge that due to time constraints, many experiments and combinations were not explored.
- Manual Annotation Gap: A key limitation is that they did not fine-tune Mistral-7B on the manually annotated data collected through crowd-sourcing. They express a desire to complete this experiment in the future.
- Synthetic Data Quality: While effective, the quality of synthetic data can always be improved, and further research into prompt engineering and generation strategies is warranted.
- Language Coverage: While the task covered 10 languages, the team focused their LLM fine-tuning efforts on Spanish, English, and Portuguese due to time and familiarity constraints. Expanding to other languages would be a natural next step.
Summary updated