Personalized Text Retrieval for Learners of Chinese as a Foreign Language

Chak Yan Yeung, John LeeCOLING 2018

TL;DR

This paper introduces a personalized text retrieval system that helps Chinese language learners find reading materials tailored to their vocabulary complexity. By training a custom complex word identification model for each user, the system effectively identifies simpler texts for beginners and more challenging ones for advanced learners, bridging the gap between overly easy and overly difficult content.

What Problem Does This Paper Address?

Learning a foreign language, especially one as complex as Chinese, benefits greatly from extensive reading. However, finding the "sweet spot" of reading material is challenging. Texts that are too difficult overwhelm learners, while those that are too easy fail to advance their skills. Existing solutions like graded readers offer limited options and coarse proficiency levels. Language teachers struggle to personalize recommendations for every student. This paper tackles the problem of automatically identifying reading materials that match a learner's specific vocabulary knowledge, aiming to strike a balance between comprehension and challenge.

Methods

The core of the proposed system is a personalized text retrieval algorithm with three main components:

  1. Training Set Creation: The system uses a graph-based active learning model to select a small, informative set of words (50 words, following prior work) for the user to rate. This approach assumes words with similar frequency ranks are likely to be known by learners with similar vocabulary profiles.
  2. Complex Word Identification (CWI): After the user rates their knowledge of the selected words on a five-point scale, the system trains a personalized CWI model. Unlike previous approaches that often use a binary "complex/non-complex" classification, this work extends it to a three-way classification: "non-complex," "challenging," and "complex." This finer granularity allows for more nuanced text selection. Features used for CWI include word difficulty levels from HSK and TOCFL guidelines, character-level HSK scores, and word frequencies from general and learner corpora.
  3. Text Retrieval: The CWI model is then applied to find texts that meet a user-specified minimum proportion of understandable words (set to 80% in experiments). The system retrieves texts that satisfy this comprehension threshold and ranks them based on the proportion of "challenging" words, prioritizing texts that offer a good learning opportunity without sacrificing readability.

Datasets

  • CWI Training and Test Sets: The study utilized the same training set of 50 words and test set of 550 words as a previous work (Lee and Yeung, 2018), annotated by eight Chinese as a Foreign Language (CFL) learners. These learners were divided into "Low-Proficiency" (knowing <150 words out of 550) and "High-Proficiency" groups.
  • Text Retrieval Corpus: A collection of texts was compiled from Chinese Wikipedia, the Chinese Internet Corpus (Sharoff, 2006), and Duan Mei Wen. Approximately 38,881 texts were used, with each text excerpted to a length of 200-225 characters.

Key Insights and Ideas

  • Beyond Binary Complexity: The move from binary (complex/non-complex) to three-way (non-complex, challenging, complex) CWI is crucial. It acknowledges that words can be understood with some effort ("challenging") without being completely unknown ("complex"), which is vital for effective language learning.
  • Context Matters for Comprehension: The study highlights a significant discrepancy between how words are rated in isolation (for CWI) and in context (during text retrieval). Words rated as "challenging" or "complex" in isolation were often understood as "non-complex" by learners when encountered within a text. This suggests that context significantly aids comprehension and word guessing.
  • Personalization is Key: The effectiveness of personalized CWI models is demonstrated, showing that individual vocabulary knowledge is a critical factor in selecting appropriate reading material.
  • Opposite Effects for Different Proficiency Levels: The system's retrieval strategy has opposite but beneficial effects for different learner groups. For high-proficiency learners, it finds more challenging texts, while for low-proficiency learners, it successfully identifies more readable texts, avoiding the baseline's tendency to retrieve overly difficult material for beginners.

Key Results

  • CWI Performance: The best CWI models achieved accuracies around 64-65% for all learners. Performance was generally better for low-proficiency learners (around 70%) than for high-proficiency learners (around 58%).
  • Text Retrieval for High-Proficiency Learners: The WordList model and Learner+WordList model retrieved texts with a lower proportion of non-complex words (90.18% and 87.87%) compared to the random baseline (95.36%), indicating they found more challenging material suitable for advanced learners.
  • Text Retrieval for Low-Proficiency Learners: The WordList model and Learner+WordList model retrieved texts with a higher proportion of non-complex words (80.60% and 86.26%) compared to the random baseline (72.91%), successfully finding more readable texts for beginners and meeting the 80% comprehension threshold.
  • Contextual Understanding: In the text retrieval evaluation, a significant portion of words initially labeled as "challenging" or "complex" by the CWI models were re-classified as "non-complex" by learners when they appeared in context (e.g., 68% of "challenging" words and 36.67% of "complex" words were re-classified).

Limitations and Future Work

  • Limitations:
    • The study acknowledges that the CWI models' performance is not perfect, and the discrepancy between isolated word ratings and contextual understanding is a significant factor.
  • The manual effort required for users to rate words, even for a small set, can be a barrier.
  • Future Work:
    • The authors suggest further exploring how context influences word difficulty perception.
  • Investigating more efficient ways to build personalized CWI models and reduce user annotation burden would be beneficial.
  • Expanding the CWI to include more nuanced categories of vocabulary knowledge could further refine text retrieval.

Summary updated

BibTeX
@inproceedings{yeung2018personalized,
  author = {Chak Yan Yeung and John Lee},
  title = {{Personalized Text Retrieval for Learners of Chinese as a Foreign Language}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure