TL;DR
This paper presents a system for identifying complex words in text, a crucial step for improving readability. The authors developed a machine learning model using a Random Forest Regressor trained on a novel English dataset with Likert-scale complexity ratings, achieving competitive results in the SemEval-2021 Task 1.
What Problem Does This Paper Address?
The paper tackles the challenge of Complex Word Identification (CWI). CWI aims to pinpoint words within a text that are difficult for a specific target audience to understand. This is vital for applications like text simplification, which seeks to make content more accessible to individuals with varying literacy levels, non-native speakers, or those with cognitive disabilities. While CWI has been an active research area, the authors highlight the need for robust systems that can accurately predict word complexity in context, especially with new datasets and approaches.
Methods
The core of the system is a Random Forest Regression algorithm. This supervised learning approach was chosen for its ability to handle a variety of features and its robustness. The authors extracted a comprehensive set of 15 word-level features to represent each target word and its context. These features include:
- Lexical features: Absolute and relative word frequency, word length, number of syllables.
- Syntactic features: Part-of-Speech (POS) tag.
- Positional features: Target word position in the sentence, number of words in the sentence.
- Contextual features: Relative frequency and length of the preceding and succeeding words.
- Lexical diversity measures: Measure of Textual Lexical Diversity (MTLD-diversity).
- WordNet-based features: Number of synonyms, hyponyms, and hyperonyms.
For multi-word expressions, similar features were adapted. Before training, numerical features were scaled to a standard range, and a polynomial transformation (degree 2) was applied to create new features. The dataset was split into 90% for training and 10% for testing.
Datasets
The study utilized data from the SemEval-2021 Task 1: Lexical Complexity Prediction (LCP) shared task. The training and evaluation corpora were provided by the task organizers. This dataset is an augmented version of the CompLex corpus and includes sentences from three different sources: the Bible, Europarl, and biomedic texts. Each instance consists of a sentence, a targeted token (word), and its complexity level rated on a Likert scale. This new dataset is described as a multidomain English dataset.
Key Insights and Ideas
- Feature Engineering is Key: The paper strongly suggests that a richer set of word-level and contextual features significantly improves classification precision. The inclusion of features beyond simple frequency, like POS tags and WordNet relations, proved beneficial.
- Random Forest's Efficacy: The Random Forest Regression model demonstrated its effectiveness in handling the diverse feature set and predicting continuous complexity values.
- Context Matters: Features that capture the surrounding words (frequency and length of neighbors) were important, indicating that a word's complexity is not solely an intrinsic property but also depends on its usage.
- Simplicity Can Be Competitive: Despite using a relatively straightforward approach with a Random Forest and a curated set of features, the system achieved competitive results, suggesting that a well-engineered feature set can be highly effective.
Key Results
The authors reported their best results on the trial and simulated data using 150 nodes and 7 selected features (abs_frecuency, rel_frecuency, lenght, number_syllables, token_position, number_synonyms, Part_of_speech). These yielded a Mean Absolute Error (MAE) of 0.07347, Mean Squared Error (MSE) of 0.00938, and Root Mean Squared Error (RMSE) of 0.096871.
In the official SemEval-2021 competition for the evaluation corpus of simple words, their system achieved an MAE of 0.0875, MSE of 0.0131, and R-squared of 0.1930. While not winning the top spot (the first-place winner had an MAE of 0.0609), their performance was considered competitive given the large number of participants and the small difference in error metrics.
Limitations and Future Work
The authors acknowledge that their approach is relatively simple. As future work, they plan to:
- Perform error analysis: To identify specific weaknesses of their approach and characterize instances where the system performs poorly.
- Analyze multi-word scenarios more deeply: The paper mentions that multi-word expressions were handled, but a more in-depth analysis is foreseen.
- Explore more complex models: While Random Forest was effective, investigating more advanced deep learning architectures could potentially yield further improvements.
Summary updated