SB@GU at the Complex Word Identification 2018 Shared Task

David Alfter, Ildikó PilánNAACL 2018

TL;DR

This paper presents SB@GU's participation in the 2018 Complex Word Identification (CWI) shared task, focusing on identifying words that might be difficult for language learners or individuals with reading difficulties. They found that simple n-gram models performed surprisingly well, even on par with more complex feature-engineered systems, particularly for non-English languages.

What Problem Does This Paper Address?

The core problem is making text more accessible. Complex Word Identification (CWI) aims to automatically detect words that are hard to understand. Once identified, these words can be replaced with simpler synonyms, a process crucial for text simplification systems. This is particularly beneficial for language learners and individuals with reading challenges, as readable text facilitates information processing. The 2018 shared task specifically focused on CWI for multiple languages, presenting a challenge in adapting systems across linguistic boundaries and dealing with varying annotation schemes.

Methods

SB@GU's approach involved two main strategies:

  • Feature-Engineered System (English): For English, they built a system leveraging a rich set of features. These included:
  • Count and Word Form Features: Length, syllable count, character bigrams, and presence in specific lists (e.g., Ogden list, Academic Word List).
  • Morphological Features: Part-of-speech and suffix length.
  • Semantic Features: WordNet synsets, hypernyms, hyponyms, and sense ID.
  • Context Features: Topic distributions and word embeddings.
  • Psycholinguistic Features: N-Watch features (e.g., corpus frequencies, reaction times). They also developed a strategy to handle multi-word expressions (MWEs) by summing or averaging feature vectors of their constituent words.

  • Simpler Systems (German, Spanish, French): For the non-English languages, they opted for a simpler approach based on character n-gram models and word length. This was motivated by initial findings that these simpler models could achieve comparable results to more complex systems.

They also explored different classifiers, including Random Forest, Extra Trees, and neural networks, and employed feature selection techniques to identify the most informative features.

Datasets

The paper utilizes data from the Complex Word Identification (CWI) 2018 Shared Task. The datasets included:

  • English: Annotated by native and non-native speakers.
  • Spanish: Annotated by native and non-native speakers.
  • German: Annotated by native and non-native speakers.
  • French: No training data was provided for this sub-task.

The paper also references previous work using datasets from 2016, and mentions the use of external resources like Wikipedia dumps for language modeling and various lexical resources (e.g., Academic Word List, CEFR-graded lists).

Key Insights and Ideas

  • Simplicity Wins (Sometimes): The most striking insight is that simple character n-gram models, combined with word length, performed remarkably well for German, Spanish, and French, often matching or exceeding more complex, feature-engineered systems. This suggests that for certain languages and tasks, a less is more approach can be highly effective.
  • Genre Matters: The performance of the English system improved when trained on genre-specific data (News, WikiNews, Wikipedia) compared to a single, genre-agnostic model. This highlights that the concept of "complex word" can be influenced by the domain of the text.
  • Context Vectors Don't Always Help: Despite the intuition that context is crucial for word complexity, the authors found that word embedding context vectors did not contribute significantly to their English CWI models after feature selection. This might be due to the nature of the features or the limited data.
  • Contradictory Annotations: Representing isolated words as vectors led to a notable percentage (5%) of contradictory data points, where the same word representation received different complexity labels. This points to inherent ambiguities in annotating word complexity.
  • Language Relatedness: Using a Spanish classifier for French data yielded better results than using a combination of classifiers from different language families. This supports the idea that closely related languages can benefit from shared models, which is particularly relevant for low-resource languages.

Key Results

  • English: The best English system was an Extra Trees classifier with 1000 estimators and a reduced feature set, achieving F1 scores of 0.8325 (News), 0.8031 (WikiNews), and 0.7812 (Wikipedia) on the test set.
  • German: The best classifier achieved an F1 score of 0.7427 on the test set.
  • Spanish: The best classifier achieved an F1 score of 0.7281 on the test set.
  • French: The best classifier achieved an F1 score of 0.6266 on the test set.
  • Ranking: While their English systems didn't rank highly (13th, 13th, and 6th), their German and French systems achieved impressive ranks of 2nd and 3rd, respectively.
  • Native vs. Non-Native: Training separate classifiers for native and non-native speakers did not yield significant improvements in accuracy (around 75% for all configurations).
  • 2016 vs. 2018 Data: Systems trained on one year's data performed worse when applied to the other year's data, suggesting a dependency on the specific annotation and task setup.

Limitations and Future Work

  • Generalizability: The study raises questions about the generalizability of CWI systems across different datasets, annotations, and task variations, as evidenced by the performance drop when applying models trained on 2016 data to 2018 data and vice-versa.
  • Neural Network Performance: The authors note that their neural network models, while tested, did not perform as well as Extra Trees. They attribute this to potential issues like insufficient data, suboptimal hyperparameter tuning, or the nature of the features used.
  • Context Feature Underutilization: The fact that word embedding context features were not selected by their feature selection method suggests that either these features are not as informative for CWI as initially thought, or that the current feature selection method is not capturing their utility effectively.
  • Future Work: The authors suggest exploring more sophisticated ways to handle multi-word expressions and investigating the genre-dependency of complexity further. They also highlight the potential of leveraging related languages for low-resource scenarios.

Summary updated

BibTeX
@inproceedings{alfter2018sb,
  author = {David Alfter and Ildikó Pilán},
  title = {{SB@GU at the Complex Word Identification 2018 Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure