TL;DR
This paper presents the NILC team's approach to the Complex Word Identification (CWI) task at CWI 2018. They explored three distinct methods: traditional feature engineering, shallow neural networks with word embeddings, and a deep LSTM language model. Their findings suggest that while feature engineering yielded the best results for binary classification, deep learning methods like LSTMs show significant promise for probabilistic CWI.
What Problem Does This Paper Address?
Text simplification aims to make complex texts easier to understand, and a crucial step in this process is identifying "complex words" that might be difficult for non-native speakers or those with lower reading proficiency. However, determining word complexity is not straightforward. It's not just about word length or frequency; context, semantic nuances, and even psycholinguistic factors play a role. The challenge lies in building systems that can accurately flag these complex words while ensuring that any proposed simplifications preserve the original meaning and grammatical correctness of the sentence. This paper focuses on the "Complex Word Identification" (CWI) task, specifically for English, which involves predicting whether a given word in a sentence is likely to be understood by a non-native English speaker.
Methods
The NILC team tackled the CWI task using three main approaches:
- Feature Engineering: This traditional machine learning approach involved crafting a rich set of features. These included:
- LEX features: Basic linguistic properties like word length, syllable count, number of senses (from WordNet), and hypernym/hyponym relationships.
- N-gram features: Log probabilities of n-grams containing the target word, derived from language models trained on large corpora.
-
PSY features: Psycholinguistic attributes such as familiarity, age of acquisition, concreteness, and imagery. These features were then fed into various classifiers like Linear Regression, Logistic Regression, Decision Trees, Gradient Boosting, Extra Trees, AdaBoost, and XGBoost.
-
Average Embedding Method: This approach leveraged word embeddings (specifically GloVe) to represent words. For each target word, its embedding was used. If the "complex word" was a phrase, the average of its constituent word embeddings was taken. This vector was then passed through a simple neural network (two ReLU layers followed by a Sigmoid layer) to predict complexity.
- LSTM Method: Recognizing the power of Recurrent Neural Networks for sequential data, the team employed a Long Short-Term Memory (LSTM) language model. They trained an LSTM on the One Billion Word dataset. For the CWI task, the LSTM processed words preceding the target word, the target word itself, and then outputted a representation that was fed into a Sigmoid layer for prediction. This method is a form of transfer learning, using a pre-trained language model to capture contextual information.
Additionally, they explored a Sent2Vec method using Skip-thought vectors, which generated sentence embeddings, but this yielded poorer results on the development set. Finally, they implemented Ensembles by combining their best systems (Feature Engineering, MLP Average Embedding, and LSTM) using majority voting for binary classification and stacking for probabilistic classification.
Datasets
The CWI 2018 English monolingual track utilized three datasets: News, WikiNews, and Wikipedia. The distribution of these datasets into training, development, and test sets is provided in Table 1.
For training language models, they used two additional corpora:
- BookCorpus: A collection of 11,038 free books.
- One Billion Word dataset: A large benchmark for language modeling.
Key Insights and Ideas
- Feature Engineering Still Holds Its Ground: Surprisingly, the meticulously engineered features, especially when combined with a powerful classifier like XGBoost, delivered the best performance for the binary classification task. This highlights that well-crafted features can still be highly competitive against deep learning methods.
- Context is King for Probabilistic CWI: The LSTM model, by its nature, excels at capturing contextual information. This proved crucial for the probabilistic classification task, where it outperformed other individual methods by a significant margin.
- Deep Learning's Potential for Nuance: While feature engineering was king for binary classification, the paper hints that deep learning methods, particularly LSTMs, are better suited for capturing the subtle nuances required for probabilistic predictions of word complexity.
- Ensembling Offers Incremental Gains: Combining models through voting and stacking provided some improvements, especially for probabilistic classification, but didn't consistently surpass the best individual feature engineering approach for binary classification.
Key Results
- Binary Classification: The Feature Engineering method, specifically using an XGBoost classifier, achieved the best ROC-AUC on the development set (0.91). The majority voting ensemble also performed very well, slightly outperforming individual methods on two test sets.
- Probabilistic Classification: The LSTM method was the strongest individual system by a large margin. However, the stacking ensemble performed slightly better than individual methods on some datasets, though it was not always superior to the standalone LSTM for the News test set. Table 2 shows F1 scores for binary classification, with XGBoost Linguistics achieving the highest F1 (0.8606) on the News dataset. Table 3 shows MAE for probabilistic classification, with the LSTM Transfer Learning achieving the lowest MAE (0.0588) on the News dataset.
Limitations and Future Work
- Limitations: The paper notes that the thresholds for classification were found for the entire training set, not for individual subsets, to avoid overfitting. While this is a good practice, it might mean that optimal thresholds for specific datasets were not utilized. The performance of the Sent2Vec method was poor on the development set, indicating it was not a strong contender for this task.
- Future Work: The authors express interest in exploring more advanced neural language models, such as character embeddings, bidirectional LSTMs, and other transfer learning techniques. This suggests a continued focus on leveraging the power of deep learning for complex NLP tasks like CWI.
Summary updated