Garuda & Bhasha at SemEval-2016 Task 11: Complex Word Identification Using Aggregated Learning Models

Prafulla Choubey, Shubham PateriaSemEval @ NAACL 2016 · DOI

TL;DR

This paper tackles the challenge of automatically identifying complex words in text, a crucial step for improving text simplification systems. The authors explore various combinations of Support Vector Machines (SVM) and Decision Trees (DT) with extensive feature engineering, ultimately finding that simpler models with fewer features can outperform complex ensembles.

What Problem Does This Paper Address?

Lexical simplification (LS) aims to make text easier to understand by replacing complex words with simpler alternatives. A critical bottleneck in LS systems is accurately identifying which words are "complex" in the first place. Errors in this Complex Word Identification (CWI) step directly impact the quality of the simplified text. Previous approaches have relied on lexicons, frequency thresholds, and basic features like word length. This paper addresses the need for robust CWI systems by investigating more sophisticated machine learning models and feature sets, aiming to improve both precision and recall in identifying complex words.

Methods

The authors propose a three-stage pipeline for CWI: pre-processing, classification, and post-processing.

  • Pre-processing: This stage uses rule-based checks to quickly classify words. Named entities and non-English words are deemed non-complex and complex, respectively. Short words (length < 2) and specific Part-of-Speech (POS) tags (CD, DT, TO) are also classified as non-complex.
  • Classification: This is the core of the system, employing aggregated learning models. They experimented with:
  • HSVM&DT (Hybrid SVM & Decision Tree): This model uses an ensemble of 5 SVMs and 5 DTs. SVMs classify words, and DTs then assess the correctness of the SVM predictions. Only SVM predictions deemed correct by the DTs are used for the final classification.
  • SVMPP (SVM with Post-Processing): This model trains 20 separate SVMs, one for each annotator in the training data. It uses coefficients derived from True Positives, False Positives, True Negatives, and False Negatives to weigh the SVM outputs.
  • Individual SVM and DT Classifiers: Standard SVM and DT models were also trained and evaluated.
  • Post-Processing: This stage aims to correct highly probable miss-classifications using character bigram frequencies and word occurrence frequencies.

The feature sets used include character n-grams, word frequencies (including surrounding words), POS tags, word length, ratio of vowels/consonants, and WordNet synonym information.

Datasets

The paper uses the SemEval-2016 Task 11 CWI dataset for training and evaluation. They also mention using the Brown corpus and the Penn Treebank for POS tagging and the WordNet lexical database.

Key Insights and Ideas

  • Feature Engineering is Crucial but Tricky: While extensive feature engineering was performed, the paper suggests that an overabundance of features can sometimes obscure the impact of simpler, more effective ones like word frequency.
  • Ensemble Complexity vs. Performance: The most complex ensemble model (HSVM&DT) performed worse than simpler individual classifiers, highlighting that aggregation doesn't always guarantee better results. The authors attribute this to overlapping decision boundaries among the SVMs.
  • Simpler Models Can Be Effective: The paper hints that simpler models, particularly DTs, might be more effective for CWI, especially when trained on carefully selected features.
  • Word Frequency as a Strong Indicator: The authors acknowledge that word frequency is a highly confident feature for determining word complexity, a finding consistent with prior work.

Key Results

The paper evaluated four systems: SVM&DT, SVMPP (from team Garuda), and SVM, DT (from team Bhasha).

  • Overall Performance: None of the submitted systems achieved top performance. They were ranked 31st, 38th, and 41st out of 52 systems based on the G-score.
  • HSVM&DT: This system performed particularly poorly, ranking 48th.
  • SVMPP: Achieved a G-score of 0.546.
  • SVM (Bhasha): Achieved a G-score of 0.508.
  • DT (Bhasha): Achieved a G-score of 0.529.

The authors note that their SVM and SVMPP systems were comparable to other SVM-based submissions, but their DT system did not perform as well as other DT or RFC-based systems.

Limitations and Future Work

  • Feature Selection: The authors acknowledge that improper feature selection was a major reason for their systems' suboptimal performance, with too many features potentially "shadowing the impact of frequency feature."
  • Ensemble Diversity: The failure of the HSVM&DT model is attributed to a lack of diversity in the classifiers, leading to overlapping decision boundaries.
  • Future Work: The authors plan to focus on developing a suitable feature selection method for CWI. They also intend to continue working on ensemble classifiers, specifically exploring models similar to HSVM&DT but with improved diversity among constituent classifiers. They also suggest that further exploration of the HSVM&DT model's potential is warranted.

Summary updated

BibTeX
@inproceedings{choubey2016garuda,
  author = {Prafulla Choubey and Shubham Pateria},
  title = {{Garuda \& Bhasha at SemEval-2016 Task 11: Complex Word Identification Using Aggregated Learning Models}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1156},
}
Figure