Complex Word Identification Based on Frequency in a Learner Corpus

Tomoyuki Kajiwara, Mamoru KomachiNAACL 2018

TL;DR

This paper introduces a simple yet effective system for identifying complex words in text, a crucial step for making content accessible to language learners. By leveraging word frequency in learner corpora alongside native speaker corpora, their system achieved top performance in the CWI Shared Task 2018, demonstrating the significant value of learner-specific language data.

What Problem Does This Paper Address?

Lexical simplification aims to make text easier to understand, particularly for language learners. A key step in this process is "Complex Word Identification" (CWI), which involves pinpointing words that might be difficult for a non-native speaker. Previous research has largely relied on word frequency in corpora written by native speakers (like Wikipedia) to gauge word difficulty. However, language learners often use simpler vocabulary than native speakers. This paper addresses the gap by investigating whether word frequency in a learner corpus is a more effective indicator of word complexity for CWI than traditional native-speaker corpora.

Methods

The authors developed systems for the CWI Shared Task 2018 using random forest classifiers and regressors. Their approach is characterized by its simplicity, relying on a set of straightforward features:

  • Lexical Features:
    • Number of characters in the target word/phrase.
  • Number of words in the target phrase (since longer phrases tend to be less frequent and thus potentially more complex).
  • Frequency-Based Features:
    • Frequency of the target word/phrase in various corpora:
  • Native speaker corpora: Wikipedia, WikiNews.
  • Learner corpus: Lang-8 corpus.
  • Probability of the target word/phrase in these corpora.

The core novelty lies in the inclusion and evaluation of the Lang-8 corpus, a learner corpus, as a feature source. They explored different combinations of hyperparameters for the random forest models.

Datasets

The systems were trained and evaluated on datasets from the CWI Shared Task 2018. This task involved a multilingual dataset with data from:

  • English: News (professionally written), WikiNews, and Wikipedia.
  • Spanish: Wikipedia.
  • German: Wikipedia.
  • French: Wikipedia.

The Lang-8 corpus was used as a learner corpus for feature extraction. The paper notes that the CWI Shared Task 2018 used a multilingual dataset with multiple annotators per instance, a contrast to the first shared task which focused solely on English and had single annotators for test instances.

Key Insights and Ideas

  • Learner Corpus is Key: The most significant finding is that word frequency in a learner corpus (Lang-8) is a highly valuable feature for Complex Word Identification, often outperforming or complementing frequency data from native speaker corpora. This validates the intuition that learners' language use patterns are distinct and informative for assessing difficulty.
  • Simplicity Wins: Despite using relatively simple features (length and frequency), their random forest models achieved competitive, and in many cases, top-tier results across multiple tracks of the shared task. This suggests that a well-chosen set of basic features can be very powerful.
  • Frequency vs. Probability: Ablation studies showed that while both frequency and probability features are important, probability features tend to be slightly more impactful. However, using both together yields the best performance.
  • Wikipedia's Limited Impact: Interestingly, the largest native speaker corpus, Wikipedia, did not contribute significantly to performance in ablation studies when compared to other corpora. This further emphasizes the unique value of the learner corpus.

Key Results

The TMU systems performed exceptionally well in the CWI Shared Task 2018:

  • Binary Classification Task (Macro-averaged F1-score): They ranked first in Spanish and German, and second in French.
  • Probabilistic Classification Task (MAE): They ranked first in Spanish, German, and English news tracks, and second in the English WikiNews track.
  • Out of 12 tracks, their simple systems achieved the best performance on 5 tracks.

Limitations and Future Work

  • Limitations: The paper acknowledges that their systems are "simple." While effective, they might not capture all nuances of word complexity. The reliance on frequency might miss words that are rare but conceptually simple, or common but semantically complex.
  • Future Work: The authors implicitly suggest exploring more sophisticated features or models that can further leverage the insights gained from learner corpora. Further investigation into why certain corpora are more effective than others could also be a fruitful direction. The paper also highlights the need for more research into what constitutes a "useful corpus" for CWI.

Summary updated

BibTeX
@inproceedings{kajiwara2018complex,
  author = {Tomoyuki Kajiwara and Mamoru Komachi},
  title = {{Complex Word Identification Based on Frequency in a Learner Corpus}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure