TL;DR
This paper tackles the challenge of automatically identifying "complex" words for non-native English speakers. The authors propose a voting ensemble of eight classifiers, leveraging lexical, size, and vocabulary features, which achieves competitive results on multiple datasets, outperforming many individual models. This work highlights the power of combining multiple, simpler models to achieve superior performance in complex linguistic tasks.
What Problem Does This Paper Address?
The paper addresses the problem of Complex Word Identification (CWI), a crucial sub-task for Lexical Simplification. Lexical Simplification aims to make texts more accessible by replacing difficult words with simpler synonyms. This is particularly important for audiences like non-native English speakers, language learners, young readers, and individuals with language disabilities. The core challenge is to computationally determine whether a given word, within its context, is likely to be difficult for a specific target audience. Unlike previous work, this paper considers multi-word expressions as target words and contexts that can span multiple sentences.
Methods
The authors employ a voting ensemble of eight calibrated classifiers to make the final decision on word complexity. These classifiers are trained using a combination of lexical, size-based, and vocabulary-based features.
- Lexical Features: These are derived from WordNet and include:
- Degree of Polysemy (DP): The number of senses a word has.
- Hyponym/Hypernym Tree Depth (HoTD/HeTD): The depth of a word in WordNet's hierarchical structure, indicating its generality.
- Holonym/Meronym Count (HC/MC): Features related to a word's part-whole relationships.
- Verb Entailments (VE): For verbs, the number of actions they entail.
- Size-based Features: These include properties of the target word itself, such as word length, vowel count, syllable count, and word count.
- Vocabulary-based Features: These check for the presence of the target word in curated word lists, like Ogden's Basic/Frequent Lexicons and Barron's GRE Word List, which are indicative of vocabulary difficulty.
The eight chosen classifiers (Random Forest, J48 Decision Tree, Logistic Model Tree, REP Tree, Random Tree, SVM, PART, and JRip Rules Tree) were selected based on their performance in a 10-fold cross-validation on the training data, specifically achieving an F1-Score above 0.70 for the "complex" class.
For tie-breaking situations (where four classifiers predict "complex" and four predict "simple"), a GloVe pre-trained word embedding-based classifier is used. This classifier identifies the most similar words to the target word's constituents and flags it as complex if any similar word was tagged as complex during training.
Datasets
The study utilizes datasets from the NLP BEA's CWI shared task at NAACL 2018. The English monolingual dataset comprises sentences from:
- NEWS: Articles written by professional journalists.
- WIKINEWS: Articles written by less experienced writers.
- WIKIPEDIA: Articles from Wikipedia.
These datasets were annotated by both native and non-native English speakers, with a word being marked as complex if even one annotator found it difficult.
Key Insights and Ideas
- Ensemble Power: The core idea, "The whole is greater than the sum of its parts," is strongly supported. Individual classifiers, while decent, are significantly outperformed by the ensemble, demonstrating the benefit of combining diverse predictive models.
- Feature Importance: Size-based features (word length, vowel count, syllable count, word count) emerge as the most significant indicators of complexity, followed by the presence of words in the Barron's GRE Word List, which signifies advanced vocabulary.
- Contextual Complexity: The inclusion of multi-sentence contexts and multi-word expressions acknowledges the nuanced nature of word complexity, moving beyond simple sentence-level analysis.
- Tie-breaking Strategy: The use of GloVe embeddings for tie-breaking adds a sophisticated layer to the ensemble, leveraging semantic similarity to resolve ambiguous cases.
Key Results
The proposed system achieved competitive results across the three datasets:
- WIKINEWS: 4th place
- WIKIPEDIA: 5th place
- NEWS: 6th place
The F1-Scores ranged from 0.777 to 0.855 for the different datasets. The system's performance was within 0.026 to 0.042 percent of the best-performing model in the shared task. Notably, out of 4252 test points, only 173 (approximately 4.06%) resulted in a tie, which the ensemble and tie-breaker successfully handled.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the majority class was "non-complex," which can sometimes skew classifier performance.
- While the ensemble is effective, a small percentage of predictions still resulted in ties (4.06%), requiring the tie-breaker.
- Future Work: The authors plan to enhance the system by incorporating additional features such as:
- Parts of Speech (POS) tags.
- Named Entity Recognition (NER) tags.
- Word position features within the sentence.
Summary updated