CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble-Based Voting

Sian Gooding, Ekaterina KochmarNAACL 2018

TL;DR

This paper presents the winning systems from the CWI Shared Task 2018, achieving top performance in identifying complex words in text. Their ensemble-based voting approach, combining multiple classifiers and leveraging a rich set of features, proved highly effective across various text genres.

What Problem Does This Paper Address?

Reading comprehension is crucial, and complex technical terms can be a significant barrier. Complex Word Identification (CWI) aims to automatically flag these difficult words so they can be simplified or explained, aiding readers with varying levels of language proficiency. While CWI has been explored, challenges remain in creating robust systems that perform well across different text types and in accurately identifying complexity in context. This paper addresses the need for effective CWI systems by participating in and winning a shared task designed to benchmark these approaches.

Methods

The CAMB team employed an ensemble-based voting strategy for their winning systems. For binary classification (complex vs. simple), they found that an ensemble combining AdaBoost and Random Forest classifiers yielded the best results, particularly for the WIKINEWS dataset. For probabilistic classification (scoring complexity on a scale), they utilized a Linear Regression algorithm.

Their feature set was extensive, drawing from previous work and incorporating new elements:

  • Word N-gram and POS: Capturing word patterns and grammatical roles.
  • Lexical Features: Word length, syllable count, and WordNet-based features (number of senses, hypernyms, hyponyms).
  • Dependency Parse Relations: Information about how words relate to each other syntactically.
  • Lexicon-Based Features: Presence of words in curated lists like SubIMDB, Simple Wikipedia, and Ogden's Basic English.
  • Word Frequency: Estimated using Google N-grams.
  • MCR Features: Psycholinguistic features like familiarity, imageability, and concreteness from the MRC Psycholinguistic Database.
  • CEFR Levels: Derived from the Cambridge Advanced Learner Dictionary.

They also developed specific approaches for phrase complexity prediction, including a method that leverages the word-based CWI classifier and an n-gram classifier.

Datasets

The CWI Shared Task 2018 utilized the CWIG3G2 dataset, which comprises texts from three genres:

  • NEWS: Professionally written news articles.
  • WIKINEWS: Amateurishly written news articles.
  • WIKIPEDIA: Wikipedia articles.

This dataset includes annotations for both words and phrases, collected from native and non-native English speakers. It differs from previous datasets by using a wider range of genres and by providing both binary (complex/simple) and probabilistic (score from 0.0 to 1.0) labels.

Key Insights and Ideas

  • Ensemble Power: Ensemble methods, particularly voting classifiers, are highly effective for CWI, outperforming individual models.
  • Genre Matters: Performance varies significantly across genres. NEWS data, with its more restricted vocabulary, appears easier for classifiers to learn from, while WIKIPEDIA, with its broad scope and unique words, presents a greater challenge.
  • Context is King (but hard to model): The paper acknowledges that their context-independent systems are a major source of error. The complexity of a word can heavily depend on its usage within a sentence, a factor their current models don't fully capture.
  • Annotation Nuances: The dataset reveals that word complexity is not always binary but exists on a spectrum, and annotators can disagree, especially with proper nouns where world knowledge plays a larger role.
  • Feature Importance Varies: The most informative features differ across datasets, highlighting the need for adaptable models. For instance, MCR psycholinguistic features were less beneficial for WIKIPEDIA, likely due to its sparsity of coverage in that database.

Key Results

The CAMB systems achieved impressive results in the CWI Shared Task 2018:

  • Binary Classification: They scored first in the English track across all three text genres (NEWS, WIKINEWS, WIKIPEDIA), with F1 scores of 0.8736, 0.8400, and 0.8115, respectively.
  • Probabilistic Classification: They achieved first place on WIKINEWS and WIKIPEDIA test sets and third place on the NEWS test set, with Mean Absolute Errors (MAE) of 0.0674, 0.0739, and 0.0558, respectively.

Limitations and Future Work

  • Context Independence: The primary limitation identified is the context-independent nature of their CWI systems. Future work will focus on incorporating contextual features to improve accuracy.
  • Phrase Complexity Derivation: The paper suggests further investigation into how phrase complexity can be more effectively derived from individual word complexity scores.
  • Demographic Factors: The authors believe that demographic factors (education, L1, language proficiency) are crucial for understanding word complexity and should be integrated into both data annotation and CWI detection.

Summary updated

BibTeX
@inproceedings{gooding2018camb,
  author = {Sian Gooding and Ekaterina Kochmar},
  title = {{CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble-Based Voting}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure