Modeling language learning using specialized Elo rating

Jue Hou, Maximilian W. Koppatz, José María Hoya Quecedo, Nataliya Stoyanova, Mikhail Kopotev, Roman YangarberACL 2019

TL;DR

This paper introduces a novel application of Elo ratings to model language learner proficiency and concept difficulty in an unsupervised manner. By adapting Elo for educational contexts, they can automatically assess learners and build a concept graph representing linguistic dependencies, offering a more scalable and data-driven approach to intelligent tutoring systems.

What Problem Does This Paper Address?

Intelligent Tutoring Systems (ITS) aim to provide personalized learning experiences, but a critical component is accurately assessing a learner's current proficiency. Traditional methods for modeling learner knowledge, like Bayesian Knowledge Tracing, can be complex to implement and require substantial data per learner. Furthermore, understanding the relationships between different linguistic concepts is crucial for building effective learning pathways. This paper addresses the need for a simpler, yet robust, method to dynamically assess learner proficiency and the difficulty of linguistic concepts, and to infer the structure of the domain knowledge itself from real-world educational data.

Methods

The core of this work is the adaptation of the Elo rating system, originally designed for chess players, to the domain of language learning. The authors apply Elo in two distinct assessment contexts:

  • Language Tests: In this context, a student's attempt at a multiple-choice question is treated as a "match" between the student and the question's linguistic concept. The Elo formula is modified to account for the probability of guessing in multiple-choice questions. Instead of rating individual questions, they rate the linguistic concepts themselves, using questions as representatives of these concepts.
  • Language Practice: Here, students engage with exercises based on authentic texts. The "match" is between the student and the text. The outcome is the student's percentage of correct answers for a set of exercises related to that text. The text's difficulty is estimated using a linear regression model based on lexical frequency, mean token length, and mean sentence length. Crucially, text Elo ratings are updated with respect to individual users, reflecting that a text can become "easier" for a specific learner over time.

Beyond individual ratings, the paper also introduces a method to build a concept graph. This graph represents the dependencies between linguistic concepts (e.g., one concept being a prerequisite for another). This is achieved by analyzing user mastery data across concepts and identifying patterns of consistency or inconsistency in how users master related concepts. The resulting graph is a directed acyclic graph (DAG).

Datasets

The study utilizes educational data collected from real language learners of Russian as a second language (L2) at various CEFR levels (A1-C2). This data comes from two contexts:

  • Language Tests: Approximately 600,000 test answers from about 1000 learners, with each question linked to one of 140 predefined linguistic concepts.
  • Language Practice: Student answers to exercises based on texts, along with text difficulty ratings and timestamps.

The paper does not introduce a new benchmark dataset but rather leverages existing data from their Revita ICALL system.

Key Insights and Ideas

  • Elo for Concepts and Learners: The adaptation of Elo ratings proves effective for simultaneously modeling both learner proficiency and concept difficulty, offering a more holistic view of the learning process.
  • Unsupervised Domain Modeling: The ability to infer a concept graph representing linguistic dependencies directly from learner data, without explicit expert input, is a significant step towards automated curriculum design.
  • Context-Specific Elo: The authors demonstrate that Elo can be effectively adapted to different assessment contexts (tests vs. text-based exercises), acknowledging the unique dynamics of each.
  • Text Difficulty as a Dynamic Measure: The approach of updating text difficulty ratings based on individual learner interactions highlights the subjective nature of "difficulty" in a learning context.
  • Burn-in Period for Stability: The concept of an "Elo burn-in" period, where concept ratings are stabilized before learner ratings are finalized, is crucial for obtaining reliable concept difficulty estimates.

Key Results

  • The Elo ratings for learners show a strong correlation (0.90) with expert-annotated CEFR levels, indicating the method's validity in assessing proficiency.
  • The correlation between Elo ratings from the test context and the exercise context is 0.79, suggesting that these two assessment methods capture similar aspects of learner ability.
  • Simulations indicate that a single test of 300 items is sufficient for a learner to reach their "true" rating. Similarly, practice with around 140 snippets/problem sets can lead to a stable rating.
  • The learned concept graph generally aligns with Elo ratings, showing a tendency for more complex concepts (higher Elo) to precede simpler ones, though inversions do occur.

Limitations and Future Work

  • Subjectivity of CEFR: While the correlation with CEFR is high, the authors acknowledge that CEFR ratings are subjective and that a single Elo value cannot fully capture all aspects of language proficiency.
  • Linearity vs. Graph Structure: The paper notes that while Elo ratings are linear, the concept graph is not, and reconciling these two representations is an area for future research.
  • Comparison with Other Models: The authors plan to compare their Elo-based approach with other skill-modeling systems like Glicko and TrueSkill.
  • Dynamic Concept Difficulty: The current approach yields static difficulty ratings for concepts after a burn-in period. Future work could explore dynamic concept difficulty that changes over time.
  • Full Domain Model Evaluation: Further evaluation of the concept graph's plausibility and its utility in generating learning pathways is needed.

Summary updated

BibTeX
@inproceedings{hou2019modeling,
  author = {Jue Hou and Maximilian W. Koppatz and José María Hoya Quecedo and Nataliya Stoyanova and Mikhail Kopotev and Roman Yangarber},
  title = {{Modeling language learning using specialized Elo rating}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure