UnibucKernel: A kernel-based learning method for complex word identification

Andrei Butnaru, Radu Tudor IonescuNAACL 2018

TL;DR

This paper introduces UnibucKernel, a novel kernel-based machine learning approach for identifying complex words in text. By combining diverse linguistic features with advanced kernel methods, the system achieved a strong third place in the 2018 Complex Word Identification (CWI) Shared Task, demonstrating the power of this approach for a crucial task in natural language processing.

What Problem Does This Paper Address?

Reading comprehension for non-native speakers, individuals with reading disabilities, or those with low literacy levels often hinges on understanding complex vocabulary. Text simplification efforts aim to make content more accessible, and a fundamental step in this process is accurately identifying which words are likely to be challenging. The Complex Word Identification (CWI) task specifically tackles this by predicting whether a word is "complex" or not. While previous work has explored this, the 2018 CWI Shared Task provided a standardized benchmark, and this paper presents a system designed to excel in this challenging linguistic problem.

Methods

UnibucKernel employs a two-phase pipeline: feature extraction and kernel-based learning.

Feature Extraction: The system leverages a rich set of features, categorized as:

  • Low-level features: Character n-grams (1-4 characters) are used, based on the intuition that complex words might be formed by less common n-gram combinations compared to simpler words.
  • High-level semantic features:
    • Word Embeddings: Features are derived from pre-trained word embeddings (like word2vec and GloVe). These capture semantic relationships by analyzing the context of a word within a sentence (minimum, maximum, and mean cosine similarity to other words) and by projecting word vectors into a lower-dimensional space (using PCA) and then applying a grid-based binning strategy (spatial pyramids) to represent word locations.
  • Lexical Knowledge Base: Features are extracted from WordNet, including the part-of-speech and the number of senses a word has.
  • Novel Semantic Features: The authors introduce features inspired by word sense disambiguation techniques and spatial pyramid approaches from computer vision.

Kernel-Based Learning: After feature extraction, the data is fed into a kernel-based learning framework.

  • Kernel Functions: The system uses either a linear kernel or a Radial Basis Function (RBF) kernel to compute pairwise similarities between samples.
  • Classifiers:
    • For binary classification (simple vs. complex), Support Vector Machines (SVM) are used.
  • For regression (predicting a continuous complexity score), v-Support Vector Regression (v-SVR) is employed.
  • Normalization: A pairwise kernel matrix is normalized to ensure features contribute equally to similarity calculations.

Datasets

The system was applied to the English monolingual track of the 2018 CWI Shared Task, which included three distinct data sets:

  • English News
  • English WikiNews
  • English Wikipedia

These datasets contain annotations from both native and non-native English speakers. The paper provides a summary of the number of samples in the training, validation, and test sets for each genre.

Key Insights and Ideas

  • The Power of Combined Features: The paper strongly suggests that combining low-level character-based features with high-level semantic features derived from word embeddings and lexical resources is crucial for effective complex word identification.
  • Kernel Methods Shine: The use of kernel methods, particularly the RBF kernel, proved highly effective for both classification and regression tasks, often outperforming linear kernels. This highlights their ability to capture complex, non-linear relationships in the data.
  • Word Embeddings as Semantic Anchors: Leveraging word embeddings, especially through techniques like spatial pyramid representations, provides a powerful way to encode semantic nuances and contextual differences that are indicative of word complexity.
  • v-SVR for Nuanced Complexity: The adoption of v-SVR for regression demonstrates a move towards a more granular understanding of word complexity, going beyond a simple binary classification.

Key Results

  • Competition Performance: UnibucKernel achieved a third-place ranking on the English Wikipedia data set during the 2018 CWI Shared Task.
  • Post-Competition Improvements: After fixing a bug related to NaN values in feature extraction, the system achieved improved results.
  • RBF Kernel Superiority: The RBF kernel consistently outperformed the linear kernel for both classification (higher F1-scores) and regression (lower Mean Absolute Error - MAE).
  • Regression Success: In the regression task, the system achieved better post-competition MAE scores than all competing systems on the English News and English WikiNews datasets, and the second-best score on English Wikipedia. For instance, on English News, the RBF kernel achieved an MAE of 0.0492, surpassing the competition's best of 0.0510.

Limitations and Future Work

  • Memory Constraints: The authors treated each English dataset independently due to memory limitations. They suggest that combining these datasets into a larger training set could potentially improve performance by enhancing the model's generalization capacity.
  • Feature Set Exhaustiveness: The authors acknowledge that their current feature set is "definitely far from being exhaustive" and propose the addition of more features as a key direction for future work.
  • Bug Impact: The presence of a bug in the feature extraction code during the competition likely impacted their initial performance, and the reported post-competition results represent a more accurate assessment of their method.

Summary updated

BibTeX
@inproceedings{butnaru2018unibuckernel,
  author = {Andrei Butnaru and Radu Tudor Ionescu},
  title = {{UnibucKernel: A kernel-based learning method for complex word identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure