TL;DR
This paper explores two approaches for lexical complexity prediction and simplification: one using machine-translated English texts and another using original language data. They found that while machine translation can enable multilingual processing, it introduces noise and significantly underperforms models trained directly on original language data, especially for low-resource languages.
What Problem Does This Paper Address?
Natural Language Processing (NLP) faces two intertwined challenges: Lexical Complexity Prediction (LCP), which aims to understand how difficult a phrase is to comprehend based on its words, and Lexical Simplification (LS), which seeks to make text more accessible by replacing complex words with simpler alternatives. These are crucial for improving text comprehension and accessibility across various domains. While much research has focused on English, there's a need to extend these methods to languages with less available data. This paper tackles this by investigating how machine translation can bridge this gap, allowing us to leverage existing English-centric tools for other languages.
Methods
The authors propose two distinct pipelines:
- Machine Translation (MT) Approach: This pipeline first translates texts from various languages into English. Then, it applies LCP and LS techniques that are primarily designed for English. For LCP, they use an XGBoost regressor trained on engineered features and shallow word structure features. For LS, they employ a locally run quantized LLM (OpenHermes 2.5) to generate candidate simplifications, which are then scored by complexity.
- Original Language Approach: This pipeline uses hand-crafted, language-independent features (like word length, syllable count) for LCP, trained on small trial datasets in the original languages. For LS, the same LLM is used, but this time directly on the original language texts.
A key aspect of their LCP method involves using a rich set of features, including:
- Shallow Word Structure Features: Character length, Zipf frequency, vowel/syllable counts.
- Psycholinguistic Features: Age of acquisition, concreteness, familiarity, imagability, meaningfulness, and Brown frequency, often sourced from the MRC database.
- Syntactic Features: Immediate children in dependency parses, and boolean features like "is entity."
- WordNet Features: Number of synsets, hypernyms, and hyponyms (English-only).
- External Lists: Dale-Chall list, EuroParl frequency, etc.
Datasets
The paper evaluates their approaches on datasets from the 2024 MLSP Shared Task. They specifically mention using data from the LCP2021 Shared Task for training their English models. The original language data comes from Shardlow et al. (2024b), and the translated-to-English data from Shardlow et al. (2024a).
Key Insights and Ideas
- Machine Translation as a Bridge, But a Noisy One: While translating to English allows the use of existing, well-developed NLP tools, it introduces significant noise. This noise stems from potential mistranslations, loss of context, and the introduction of multi-word expressions that aren't simple lexical substitutions.
- Original Language Data Reigns Supreme: Models trained directly on original language data, even with fewer resources, consistently outperform those relying on machine translation. This highlights the importance of preserving the nuances of the source language.
- LLMs for Simplification: Promising but Imperfect: The locally run LLM shows an ability to generate simplification candidates even for non-English languages, which is promising. However, it can hallucinate and doesn't always guarantee correct grammatical forms or contextual relevance, especially when dealing with translated text.
- Feature Engineering Still Matters: For LCP, a combination of shallow word structure and psycholinguistic features proved effective, demonstrating that traditional feature engineering still holds value, particularly for multilingual applications.
- Language Contact as a Future Direction: The observation of similar word forms across languages (e.g., Filipino "us" vs. English "us") suggests that exploring cognates and borrowings could be a fruitful avenue for future research in multilingual lexical analysis.
Key Results
- LCP Performance: The paper's LCP models trained on original language data achieved significantly better performance (higher Pearson and Spearman correlations) compared to models using machine-translated English data. For instance, Table 2 shows that for Spanish, the original language model (Run ID 1) achieved a Pearson's R of 0.5274, while the translated version (Run ID T) only reached 0.3268. This trend was observed across most languages.
- LS Candidate Generation: The LLM generated fewer empty candidates when working with English translations, but the quality and relevance of these candidates were often compromised.
- Sinhala Anomaly: Sinhala presented a unique case where word frequency was not a useful feature, and character length was the primary relevant predictor.
Limitations and Future Work
- Translation System Inaccuracies: The reliance on machine translation introduces inherent inaccuracies and limitations, especially in preserving morphological forms and context.
- LLM Limitations: The chosen LLM is English-centric, and its multilingual capabilities are limited, leading to potential errors in tense and syntactic agreement. The authors acknowledge that using closed-source models is not ideal for open research.
- Unified Framework Questionable: The authors suggest that combining LCP and LS into a single unified framework might not be the optimal approach for text comprehension.
- Future Directions: The paper points towards exploring how languages with historical contact can leverage shared lexical traits. Further research could also focus on developing more robust multilingual LLMs or improving the handling of translation noise.
Summary updated