LTG at SemEval-2016 Task 11: Complex Word Identification with Classifier Ensembles

Shervin Malmasi, Mark Dras, Marcos Zampieri

Abstract

Complex Word Identification (CWI) is a critical first step in lexical simplification. This paper details the approach taken by the LTG team for SemEval-2016 Task 11. By leveraging ensemble classification techniques—specifically bagging and threshold-based voting—and incorporating contextual features from web-scale language models, the team developed systems that effectively navigate the subjectivity of word complexity. The systems ranked 2nd and 3rd overall, highlighting the importance of robust feature engineering and ensemble strategies in handling imbalanced and noisy datasets.

Introduction

Lexical simplification aims to make text more accessible to children, language learners, and individuals with reading disabilities by replacing difficult words with simpler synonyms. The prerequisite for this process is Complex Word Identification (CWI). Identifying which words are "complex" is challenging because complexity is subjective and context-dependent. A word may be simple in one sense but complex in another (polysemy), or its difficulty may stem from an unusual usage within a specific sentence.

Methods

The LTG systems utilized Decision Tree classifiers, which were found to be more effective than Support Vector Machines or Naive Bayes for this task.

Feature Engineering

The models rely on features that capture both the word's inherent properties and its surrounding environment. Key features include word probability (frequency in general English) and word length. Crucially, the systems use contextual probabilities derived from the Microsoft Web N-Gram Service. By calculating conditional and joint probabilities for the target word and its neighbors (bigrams and trigrams), the system can detect when a common word is used in a rare or technical sense.

Contextual feature extraction using n-grams

Ensemble Strategies

Two primary ensemble configurations were submitted:

  • System 1 (Bagging): Employs Bootstrap Aggregating to create 100 randomized decision trees. It uses a training threshold where a word is labeled complex only if at least 3 out of 20 human annotators flagged it.
  • System 2 (Threshold Ensemble): Combines four classifiers, each trained on a different annotator agreement threshold (ranging from 1 to 4). The final prediction is determined by plurality voting.

Datasets

The systems were evaluated on the SemEval-2016 Task 11 dataset, which consists of a training set of 200 sentences (judged by 20 annotators each) and a test set of over 9,000 sentences (judged by only one annotator each). This discrepancy introduces a significant distribution shift, making generalization a primary challenge.

Key Insights

The Noise of Subjectivity

The authors observed that the standard "complex" label in the training data (triggered by even a single annotator's disagreement) is often noisy. By raising the agreement threshold during training, the models could filter out idiosyncratic responses and better capture general complexity trends.

The Accuracy Trap

In the test set, only 4.7% of words were labeled complex. A naive model could achieve ~95% accuracy by labeling everything as "simple." Consequently, the F-score (balancing precision and recall) and G-score are the only meaningful metrics for evaluating true performance in this imbalanced scenario.

Key Results

The LTG systems were top-tier performers in the competition, effectively identifying complex words rather than defaulting to the majority class.

Official SemEval-2016 Task 11 results

As shown in the results table, System 2 achieved an F-score of 0.312, ranking 2nd overall, while System 1 followed closely with an F-score of 0.310, ranking 3rd.

Conclusion

The primary challenge remains the extreme class imbalance in the test data, which often leads to lower precision. Future work will explore techniques for unbalanced data, such as cost-sensitive learning, and the integration of learner-specific corpora to better model the vocabulary constraints of the target audience.

Summary updated

BibTeX
@inproceedings{malmasi2016ltg,
  author = {Shervin Malmasi and Mark Dras and Marcos Zampieri},
  title = {{LTG at SemEval-2016 Task 11: Complex Word Identification with Classifier Ensembles}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1154},
}
Figure