ITEC at MLSP 2024: Transferring Predictions of Lexical Difficulty from Non-Native Readers

Anaïs TackNAACL 2024

TL;DR

This paper explores how well pre-trained models for predicting lexical difficulty in French can be transferred to a new multilingual lexical simplification task. The authors found that while direct score transfer is challenging, the models effectively preserve the ranking of word difficulty, suggesting a promising avenue for cross-lingual applications.

What Problem Does This Paper Address?

Making texts more accessible often involves simplifying complex vocabulary. This requires accurately identifying which words are difficult for readers. However, building models for this task is hampered by a lack of large, multilingual datasets. This paper tackles the challenge of leveraging existing, specialized models (trained on French) for a broader, multilingual lexical simplification task, specifically addressing the MLSP 2024 shared task. The core problem is how to adapt or transfer knowledge from a model trained on a specific language and task to a new, related task with limited or no dedicated training data for the target language(s).

Methods

The authors employed two pre-trained neural models developed by Tack (2021) for predicting lexical difficulty in French. These models were originally designed for a distinct task but were adapted for zero-shot prediction in the MLSP shared task.

  • Model 1 (Contextualized): A bidirectional Long Short-Term Memory (BiLSTM) neural network. It takes character embeddings (from a CNN) and pre-trained FastText word embeddings as input, along with learner-specific encodings (which were set to zero for this transfer task). This model predicts difficulty for words within their sentence context.
  • Model 2 (Isolated): A feedforward neural network. Similar to Model 1, it uses character and FastText embeddings but predicts difficulty for words in isolation, approximating "lexical access" difficulties.

The approach involved comparing the predictions of these two models on French test data from the MLSP task.

Datasets

  • Training Data (for pre-trained models): A French dataset comprising 262,054 binary decision annotations of perceived lexical difficulty, collected from 56 non-native French readers. This data was originally used in Tack (2021) and expanded from Tack et al. (2016a).
  • MLSP 2024 French Test Data: Provided by Shardlow et al. (2024a), this dataset contained 570 items, each with an identifier, language, sentence context, and target word. It included 560 unique word types across 191 sentence contexts. This test data was annotated by 10 non-native French speakers from various linguistic backgrounds.

Key Insights and Ideas

  • Lexical Access vs. Context Integration: The findings strongly suggest that the ground truth data primarily captures difficulties related to "lexical access" (recognizing a word's form and meaning) rather than "word-to-context integration" (understanding a word's role in a sentence). This is evidenced by the stronger correlation of isolated word predictions with the ground truth.
  • Ranking Preservation: Even though the absolute difficulty scores predicted by the models didn't perfectly align with the true complexity scores (indicated by negative R² fits), the ranking of word difficulty was well-preserved. This is a crucial insight for transfer learning, as ranking is often more important than exact scores for tasks like simplification.
  • Isolated Word Advantage: The model predicting difficulty for isolated words consistently showed a stronger correlation with the ground truth than the contextualized model, both in terms of Pearson's r and Spearman's ρ. This reinforces the idea that basic word recognition is a primary challenge for the annotators.

Key Results

  • Both models generally predicted high difficulty levels for most French test items.
  • The isolated word model (run 2) indicated a higher difficulty level than the contextualized model (run 1).
  • Pearson Correlation (r): Isolated predictions (r = .36) showed a slightly stronger correlation than contextualized predictions (r = .33).
  • Spearman Rank Correlation (ρ): Isolated predictions (ρ = .50) demonstrated a notably higher correlation than contextualized predictions (ρ = .35). This was the fourth-best Spearman rank correlation among all teams.
  • The R² metric indicated a negative fit for both models with the true complexity scores, suggesting that direct transfer of continuous scores is problematic.

Limitations and Future Work

  • Negative R² Fit: The primary limitation is the poor fit of the transferred difficulty scores with the ground truth, as indicated by negative R² values. This suggests that the models, trained for a different prediction task, don't directly map to the continuous complexity scores of the MLSP task.
  • Annotation Differences: The variation in native languages between the original training data annotators (mostly Dutch) and the MLSP test data annotators (diverse backgrounds) might have influenced the results.
  • Future Work: The authors plan to explore the implications of transferring zero-shot predictions made with pre-trained French models to other languages. This is a direct next step to test the broader applicability of their findings.

Summary updated

BibTeX
@inproceedings{tack2024itec,
  author = {Anaïs Tack},
  title = {{ITEC at MLSP 2024: Transferring Predictions of Lexical Difficulty from Non-Native Readers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure