Personalizing Lexical Simplification

John Lee, Chak Yan YeungCOLING 2018

TL;DR

This paper tackles the "one-size-fits-all" problem in lexical simplification (LS) by proposing personalized complex word identification (CWI) models. By tailoring CWI to individual users' vocabulary proficiency, they demonstrate significant improvements in LS performance, reducing unnecessary simplifications and complex words in the output.

What Problem Does This Paper Address?

Lexical simplification (LS) aims to make text easier to understand by replacing difficult words with simpler ones. However, most existing LS systems treat all users the same, assuming a universal understanding of word complexity. This is problematic because what constitutes a "complex" word is highly subjective and depends on an individual's vocabulary knowledge. A word that is challenging for a beginner might be trivial for an advanced learner. Current LS systems often fail to account for this user variability, leading to either over-simplification (making text sound unnatural or losing nuance) or under-simplification (leaving difficult words unaddressed). This paper addresses this gap by investigating how personalized Complex Word Identification (CWI) can be used to adapt LS systems to individual users' proficiency levels.

Methods

The core of this paper's approach is to personalize the Complex Word Identification (CWI) component within the LS pipeline. CWI is the crucial first step where the system identifies which words in a text are "complex" and thus candidates for simplification.

The authors propose a two-step LS algorithm:

  1. Personalized CWI for Detection: Instead of a single, user-independent CWI model, they train a CWI model specific to each user. This means the system can identify different words as complex for different users.
  2. Personalized CWI for Ranking: They also leverage personalized CWI to filter substitution candidates. If a proposed simpler word is still considered complex by the user, it can be rejected, further tailoring the output.

They explore three CWI models:

  • Baseline (nil): Always flags words as complex for detection and non-complex for ranking.
  • Oracle (gold): Represents perfect CWI, using the user's actual annotations. This serves as an upper bound for performance.
  • Graded Vocabulary List (auto): An automatic CWI model that classifies words as complex or non-complex based on predefined vocabulary lists of increasing size. This is the practical, personalized approach they advocate.

Datasets

  • User Dataset: A dataset annotated by 15 Japanese learners of English as a foreign language. Each learner rated their knowledge of 12,000 English words on a five-point scale, which was then collapsed into "complex" (scores 1-4) or "non-complex" (score 5). This dataset was used to define "Low Proficiency" (4 least proficient learners) and "High Proficiency" (4 most proficient learners) subsets.
  • BenchLS Dataset: A benchmark dataset containing 929 target words and their gold simple words, annotated by US English speakers. This dataset was adapted to create 15 personalized versions, one for each user in the user dataset. For each user, simplifications were deemed "correct" only if the target word was complex for that user and the substitution was non-complex for them.

Key Insights and Ideas

  • Subjectivity of Complexity: The paper strongly reinforces the idea that word complexity is not absolute but highly user-dependent. What is complex for one person is simple for another.
  • CWI is Key to Personalization: Personalizing the CWI step is shown to be a highly effective way to tailor LS systems. By correctly identifying what each user finds difficult, the system can make more appropriate simplifications.
  • Simple Personalization is Effective: Even a relatively simple automatic CWI model based on graded vocabulary lists can yield significant improvements, suggesting that sophisticated personalization might not always be necessary for substantial gains.
  • Balancing Precision and Accuracy: The paper highlights the trade-offs. Unnecessary simplifications (low precision) can be acceptable if they improve readability, but they can also lead to errors if the substituted word is still too complex.

Key Results

The experiments demonstrated consistent performance gains with personalized CWI:

  • Precision: Personalized CWI significantly reduced unnecessary simplifications. For example, using an oracle CWI for detection (detect=gold, rank=nil) achieved 89.95% precision, a substantial improvement over the user-independent baseline's 21.39%. The automatic CWI approach (detect=auto, rank=nil) also raised precision to 31.97%.
  • Accuracy: Personalized CWI helped the system choose non-complex substitutions, leading to higher accuracy. The oracle CWI for ranking (detect=nil, rank=gold) achieved 100% accuracy. The automatic CWI approach (detect=nil, rank=auto) improved accuracy to 94.19%, outperforming the baseline by over 4%.
  • Readability: Personalized systems generally produced better output in terms of readability, even when accepting some unnecessary simplifications. The best configuration (detect=nil, rank=auto) achieved 94.57% readability, a 3% improvement over the baseline.
  • Experiment 2 (Realistic Substitutions): The gains persisted even when using automatically generated substitutions from a state-of-the-art LS system, indicating the robustness of personalized CWI. Precision improved by up to 31% and accuracy by over 11% compared to the user-independent baseline.

Limitations and Future Work

  • Limitations:
    • The "oracle" CWI model, while showing the upper bound of personalization, is not practically achievable as it requires perfect user knowledge.
  • The automatic CWI model requires a small amount of user annotation (40 words) to personalize. While this is a small burden, it's still a requirement.
  • The paper focuses on CWI and doesn't deeply explore personalization in the substitution generation or ranking steps beyond using CWI to filter candidates.
  • Future Work:
    • The authors suggest that as CWI research produces higher-performing models, future LS systems can expect to derive even greater benefits from personalization.
  • Further exploration of personalization in other LS pipeline stages is implied.

Summary updated

BibTeX
@inproceedings{lee2018personalizing,
  author = {John Lee and Chak Yan Yeung},
  title = {{Personalizing Lexical Simplification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure