TL;DR
This paper presents MacSaar, a system for identifying complex words in English text, a crucial step for improving text readability. By leveraging Zipfian word frequencies and character-level features, MacSaar achieved a competitive 8th place in the SemEval-2016 Complex Word Identification task, demonstrating the effectiveness of these linguistic and orthographic cues.
What Problem Does This Paper Address?
The paper tackles the challenge of Complex Word Identification (CWI). CWI is a fundamental sub-task in lexical simplification, which aims to make texts easier to understand, particularly for language learners or individuals with reading impairments. The core problem is automatically distinguishing "complex" words from "simple" ones within a sentence. This isn't just about rare words; it's about words that are difficult for a target audience to comprehend. Without accurate CWI, subsequent simplification steps (like substituting complex words with simpler synonyms) would be ineffective.
Methods
The MacSaar system employs a binary text classification approach, labeling each word as either complex (1) or simple (0). Their methodology is built on two main feature sets:
- Zipfian Features: Inspired by Zipf's law, which posits an inverse relationship between word frequency and rank, the system uses both theoretical Zipfian frequency (based on word rank) and empirical "True Frequency" (calculated from corpus counts). The intuition is that complex words are generally less frequent.
- Character-based Features: To capture orthographic difficulty, the system considers:
- Word Length: Longer words are often perceived as more complex.
- Character Trigram Probability: The probability of sequences of three characters within a word, normalized to account for word length. This helps identify patterns that might be difficult for learners.
- Sentence Length and Sentence-level Trigram Probability: These features add a contextual layer, considering the overall sentence complexity.
- Ratio of Word to Sentence Orthographic Difficulty: A meta-feature combining word and sentence-level orthographic measures.
The authors experimented with three classifiers: Random Forest Classifier (RFC), Nearest Neighbor Classifier (NNC), and Support Vector Machine (SVM). For the official submission, they used RFC and SVM.
Datasets
The primary dataset used was provided by the SemEval-2016 Task 11: Complex Word Identification (CWI). This dataset consisted of:
- Training Set: 2,237 English sentences.
- Test Set: 88,221 English sentences.
The data was collected through a survey where annotators identified words they found difficult to understand. The paper notes that information about whether annotators were native English speakers was not provided. The authors also mention using the English subsection of the DSL Corpus Collection to compute word frequencies and n-gram probabilities.
Key Insights and Ideas
- Zipfian frequency and word length are strong indicators of complexity: The core assumption that less frequent and longer words are more complex holds significant predictive power.
- Orthographic features add valuable context: Character-level analysis, especially when combined with sentence context, helps capture a different dimension of word difficulty beyond just frequency.
- No single system excelled across all metrics: The SemEval results highlighted a trade-off between metrics like Accuracy, Precision, and Recall, indicating the inherent difficulty of the CWI task and the diverse strategies employed by different systems.
- Random Forest outperformed SVM: Surprisingly, the RFC classifier performed better than the SVM, despite SVM's known strengths in binary classification tasks. This suggests that the feature interactions or the specific nature of the CWI data might favor RFC's ensemble approach.
Key Results
The MacSaar system achieved competitive results in the SemEval-2016 Task 11:
- Best Performing System (RFC): Achieved 82.5% accuracy and a 27% F-Score.
- Ranking: The best MacSaar submission was ranked 8th out of 45 entries in terms of F-Measure.
- Second System (SVM): Scored 2.1 percentage points lower in accuracy and 3.0 percentage points lower in F-Score compared to the RFC system.
- The paper notes that their NNC system, tested separately, achieved 75.9% accuracy and 11% F-score, performing less well due to the training/test split.
Limitations and Future Work
- Information Content Ignored: The authors acknowledge that their approach did not incorporate "information content" as a feature, which some research suggests is a more significant predictor of word length than frequency.
- SVM Performance Analysis: They plan to investigate why the SVM classifier did not outperform the RFC, suggesting an error analysis and examination of the most informative features.
- Annotator Background: The lack of information about the annotators' native language proficiency is a limitation, as it could influence their perception of word complexity.
- Future Work: The authors suggest exploring information content and further analyzing feature importance to improve CWI systems.
Summary updated