TL;DR
This paper details the LaSTUS/TALN team's participation in the Complex Word Identification (CWI) 2018 Shared Task. They developed two systems to automatically identify complex words in English sentences: one leveraging word embeddings and another using a rich set of lexical, semantic, and contextual features. While their systems achieved reasonable performance, they highlight the ongoing challenge of effectively capturing word complexity.
What Problem Does This Paper Address?
The paper tackles the problem of Complex Word Identification (CWI). In essence, CWI aims to automatically determine if a word in a given sentence is "complex" for a specific target audience. This is a crucial sub-problem in various natural language processing tasks, including text simplification (making text easier to understand) and readability assessment (quantifying how difficult a text is to read). The challenge lies in the subjective nature of "complexity," which can depend on factors like word frequency, semantic nuance, and contextual usage, making it difficult to define and detect algorithmically. The CWI Shared Task 2018 provided a benchmark for evaluating systems on this task.
Methods
The LaSTUS/TALN team employed two distinct approaches for their CWI systems:
- Word Embedding (WE) System: This system represented words and their surrounding context using pre-trained Word2Vec embeddings. Each sentence was encoded by creating vector representations for the context before the target word, the target word itself, and the context after the target word. These representations were then fed into a binary classifier. Additionally, cosine similarity was used to capture the distance between the target word's vector and its surrounding context vectors.
- Lexical, Semantic, and Contextual (LSC) Features System: This approach relied on a comprehensive set of hand-engineered features. These features were categorized into:
- Shallow Features: Basic word properties like length and position within the sentence.
- Dependency Tree Features: Information derived from the syntactic dependency parse of the sentence, such as the depth of a word in the dependency tree and the length of its parent word.
- Corpus-based Features: Word frequencies from large corpora like the British National Corpus (BNC) and English Wikipedia, as well as the presence of words in a simple word list (Dale & Chall).
- WordNet Features: Information extracted from WordNet, including the number of synsets (senses), number of senses per synset, depth in the hypernym hierarchy, and various relation counts.
- WordNet and Corpus Frequency Features: These combined WordNet synset information with corpus frequencies. They specifically looked at the frequency of other words (lexicalizations) within a synset relative to the target word's frequency, aiming to capture semantic similarity and commonality.
Both systems utilized binary classifiers, with the Random Forest algorithm ultimately being chosen for both after evaluating several options.
Datasets
The CWI Shared Task 2018 provided the following datasets:
- Training Set: 27,299 words, each with its corresponding sentence and complexity judgments from 20 human evaluators (10 native speakers).
- Development Set: 3,328 words, also with sentences and human judgments.
- Test Set: 4,252 words with sentences.
The paper also mentions the use of external resources:
- Google News pre-trained Word2Vec model: Used for the WE system.
- British National Corpus (BNC) and 2014 English Wikipedia Dump: Used for corpus-based frequency features.
- WordNet 3.0: Used for semantic features.
- Dale & Chall list: Used to identify simple words.
- Mate dependency parser and UKB Word Sense Disambiguation algorithm: Used for processing sentences and extracting features.
Key Insights and Ideas
- Feature Engineering vs. Embeddings: The paper highlights a common trade-off in NLP: the power of rich, hand-crafted features (LSC system) versus the generality and contextual understanding of learned embeddings (WE system). Their results suggest that for certain datasets, engineered features can still be competitive or even superior.
- Context is King (but how to capture it?): Both systems emphasize the importance of context. The WE system directly models context through embeddings, while the LSC system uses features derived from syntactic dependencies and word co-occurrence. This underscores that word complexity is not an intrinsic property but is heavily influenced by its surroundings.
- Word Frequency as a Strong Baseline: The paper acknowledges that word frequency is a highly reliable predictor of word complexity, a finding consistent with previous CWI tasks. This suggests that simpler, frequency-based models can serve as strong baselines.
- Subjectivity of Complexity: The use of 20 human evaluators, including native speakers, points to the inherent subjectivity in defining "complex." This makes the task challenging and necessitates robust evaluation metrics that account for this variability.
Key Results
The LaSTUS/TALN team submitted six runs, two for each of the three subsets of the English monolingual CWI track.
- Cross-validation on training data: The Random Forest classifier achieved an F1 score of 0.810 for the WE system on the "News" dataset and 0.778 for the LSC system on the "all" dataset (combining News, WikiNews, and Wikipedia).
- Performance on shared task datasets:
- In the English News subset, their WE system achieved an accuracy of 0.8172, outperforming their LSC system (0.7785) and placing them fourth among the top three teams.
- For WikiNews, their LSC system performed better (0.7615) than their WE system (0.7374), placing them sixth.
- On the Wikipedia subset, the LSC system again outperformed the WE system (0.7414 vs. 0.6966), placing them sixth.
Overall, the WE system showed stronger performance on the "News" dataset, while the LSC system was more effective on the "WikiNews" and "Wikipedia" datasets.
Limitations and Future Work
- Limitations: The paper notes that their approaches obtained "reasonable performance in general but not in comparison with the other participating systems." This suggests there's still room for improvement in capturing word complexity. The reliance on pre-trained models and specific feature sets might limit their adaptability to diverse linguistic contexts.
- Future Work: The authors plan to explore feature selection methods (e.g., info gain) to refine their feature sets. They also express interest in applying deep learning neural networks to build more sophisticated classifiers for CWI.
Summary updated