Complex Word Identification: Convolutional Neural Network vs. Feature Engineering

Segun Taofeek Aroyehun, Jason Angel, Daniel Alejandro Pérez Alvarez, Alexander GelbukhNAACL 2018

TL;DR

This paper explores two approaches for Complex Word Identification (CWI): traditional feature engineering with tree ensembles and a novel Convolutional Neural Network (CNN) architecture. Both methods achieved comparable performance on English datasets, demonstrating the CNN's flexibility by successfully applying it to Spanish without architectural changes.

What Problem Does This Paper Address?

Complex Word Identification (CWI) is a crucial step in text simplification, aiming to identify words or phrases that might be difficult for non-native speakers or those with lower literacy. Accurately identifying these complex words is essential for downstream tasks like lexical substitution, which aims to replace complex words with simpler alternatives. The 2018 CWI shared task provided multilingual datasets to benchmark various approaches. While previous CWI tasks saw some use of neural networks, their performance was not particularly impressive. This paper investigates whether a more modern deep learning approach, specifically CNNs, can offer competitive results compared to established feature engineering techniques.

Methods

The authors compare two main approaches:

  • Feature Engineering with Tree Ensembles: This approach leverages a comprehensive set of linguistic features, including:
  • Morphological Features: Term frequency, inverse document frequency, document frequency, term document entropy, and basic word characteristics like character count, vowels, and syllables.
  • Syntactic and Lexical Features: Part-of-speech (POS) tags from OpenNLP and word senses/lemmas/hyponyms/hypernyms from WordNet.
  • Psycholinguistic and Entity Features: Familiarity, age of acquisition, concreteness, and imagery scores from the MRC psycholinguistics database, along with entity tags (organization, person, location, date, time) from OpenNLP and Stanford CoreNLP.
  • Word Embedding Distances: Cosine and Euclidean similarity between word embeddings of the target word and the context. They then used tree ensemble models (specifically, 600 models in a gradient boosted tree ensemble) from KNIME for classification.

  • Convolutional Neural Network (CNN): This is presented as a novel approach for CWI. The architecture involves:

  • Preprocessing: Removing punctuation, digits, and special characters. Words are replaced with pre-trained Word2vec (English) or fastText (Spanish) embeddings. A min-max normalization is applied to vectors. The left and right contexts are represented by a single averaged vector. A matrix is then constructed with the LC vector, target word embeddings, and RC vector, padded to a uniform size.
  • Network Architecture: A deep neural network comprising input, convolution, pooling, and fully-connected layers. The convolution layer uses varying numbers of filters and kernel sizes, followed by ReLU activation. Max pooling is used for downsampling. Three fully-connected layers are employed, with the final layer using a sigmoid activation to output a probability.
  • Training: Binary cross-entropy loss, RMSprop optimizer, and dropout regularization with a probability of 0.25 after each layer. Class weights were used to handle imbalanced datasets.

Datasets

The paper uses datasets from the CWI 2018 shared task, which provided multilingual, human-annotated data for Complex Word Identification. The languages covered are English and Spanish. For English, three genres were included: News, Wikinews, and Wikipedia. The datasets are noted to be imbalanced, with a higher proportion of simple words (negative examples) than complex words (positive examples), typically around 60% negative and 40% positive.

Key Insights and Ideas

  • CNNs are a viable alternative to feature engineering for CWI: The study shows that a CNN can achieve performance comparable to a well-engineered feature set with tree ensembles, suggesting that deep learning can effectively capture the nuances of complex words.
  • Flexibility of CNNs: The same CNN architecture, with minor adjustments for word embeddings (Word2vec vs. fastText), was successfully applied to both English and Spanish, highlighting the adaptability of the deep learning approach across languages.
  • Contextual information is key: The averaging of context word embeddings, while a simplification, proved effective in differentiating sentences with different target texts, indicating the importance of surrounding words in identifying complexity.
  • Dataset imbalance impacts performance: The authors attribute some performance issues, particularly with longer target texts, to the scarcity of such examples in the training data.

Key Results

  • English Test Set:
    • The CNN and Tree Ensemble approaches achieved comparable performance, with the difference in macro-F1 scores being within 0.01 on most genres.
  • The best CNN system ranked fifth on News, second on Wikinews, and seventh on Wikipedia.
  • On the Wikipedia test set, the best system was 0.04 below the top-ranked system.
  • Spanish Test Set:
    • The CNN-based system ranked third out of seven teams.
  • Target Text Length: A correlation was observed between target text length and model performance, with models tending to make more mistakes on longer target texts.

Limitations and Future Work

  • Limitations:
    • The performance of both models was negatively impacted by the length of the target text, which the authors attribute to the imbalanced nature of the training data (fewer examples of longer complex words).
  • Many target words were absent from the psycholinguistic database used for feature engineering.
  • Future Work:
    • Investigate the impact of domain-specific features.
  • Explore different vector operations for generating context vectors.
  • Evaluate the effect of these changes on model performance.

Summary updated

BibTeX
@inproceedings{aroyehun2018complex,
  author = {Segun Taofeek Aroyehun and Jason Angel and Daniel Alejandro Pérez Alvarez and Alexander Gelbukh},
  title = {{Complex Word Identification: Convolutional Neural Network vs. Feature Engineering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure