Optimal Hierarchical Learning Path Design with Reinforcement Learning

Xiao Li, Hanchen Xu, Jinming Zhang, Hua‐Hua Chang

Abstract

E-learning systems are capable of providing more adaptive and efficient learning experiences for students than the traditional classroom setting. A key component of such systems is the learning strategy, the algorithm that designs the learning paths for students based on information such as the students' current progresses, their skills, and learning materials. In this paper, we address the problem of finding the optimal learning strategy for an E-learning system. To this end, we first develop a model for students' hierarchical skills in the E-learning system. Based on the hierarchical skill model and the classical cognitive diagnosis model, we further develop a framework to model various proficiency levels of hierarchical skills. The optimal learning strategy on top of the hierarchical structure is found by applying a model-free reinforcement learning method, which does not require information on students' learning transition process. The effectiveness of the proposed framework is demonstrated via numerical experiments.

Introduction

While E-learning systems offer the promise of personalized education, the "brain" behind these systems—the learning strategy—is often limited. Most existing models treat skills as independent, unstructured attributes. In reality, learning is hierarchical: mastering a complex concept usually requires proficiency in several prerequisite skills.

Furthermore, most adaptive learning algorithms rely on a "transition kernel"—a mathematical map of how likely a student is to move from "not knowing" to "knowing" after seeing a specific piece of material. In practice, this kernel is almost impossible to know accurately because every student and every piece of content is different. This research presents a data-driven framework that respects skill hierarchies and finds optimal paths without needing to know the transition probabilities upfront.

Methodology

The proposed framework integrates three distinct components to optimize the learning path:

  1. Cognitive Diagnosis Models (CDMs): The system utilizes psychometric models, such as the DINA (Deterministic Inputs, Noisy "And" gate) model, to estimate a student's latent (hidden) skill levels based on their answers to test questions.
  2. Attribute Hierarchy Model (AHM): This explicitly models the dependencies between skills. For example, if Skill A is a prerequisite for Skill B, the model restricts the state space to exclude impossible scenarios. This "state space pruning" significantly reduces the complexity the AI must navigate.
  3. Q-Learning (Reinforcement Learning): The E-learning system acts as an agent that chooses actions (learning materials) to maximize a reward (student progress). Because it is model-free, the system learns the best path simply by observing the outcomes of simulated learning sessions.
The sequential student learning model showing transitions between latent skill states influenced by learning materials and resulting in observed performance.

As illustrated in the figure above, the learning process is modeled as a sequence where the latent skill state $\alpha_{i,t}$ evolves over time $t$ based on the provided learning material $l_t$, with the student's performance $\mathbf{X}_{i,t}$ serving as the observable feedback for the reinforcement learning agent.

Experimental Results

The framework was evaluated through numerical experiments simulating a curriculum with attributes like Addition and Multiplication across three proficiency levels (Beginner, Intermediate, Advanced).

  • Efficiency and Convergence: The RL method identified the optimal learning strategy after training on only 200–250 simulated students.
  • Performance Gains: In head-to-head tests, the RL method achieved a mean reward of 6.43, significantly higher than the heuristic strategy's 3.99.
  • Path Optimization: Students under the RL strategy reached full mastery in fewer steps. The mean "episode length" was 7.34 for RL versus 8.57 for the heuristic.
  • Robustness to Noise: One of the most significant findings is that the RL strategy is highly resilient. Even if the system's assessment of a student's skill is wrong 5-10% of the time, the Q-learning algorithm still identifies the optimal path.

Conclusion

This study demonstrates that combining psychometric models with model-free Reinforcement Learning can effectively navigate complex, hierarchical skill structures to help students reach mastery faster than traditional methods. By pruning the state space through hierarchical modeling, the system becomes more efficient and robust to assessment noise. Future work will focus on validating this framework in live classroom environments and exploring Deep RL techniques to handle larger curricula with hundreds of interdependent skills.

Summary updated

Abstract

E-learning systems are capable of providing more adaptive and efficient learning experiences for students than the traditional classroom setting. A key component of such systems is the learning strategy, the algorithm that designs the learning paths for students based on information such as the students' current progresses, their skills, learning materials, and etc. In this paper, we address the problem of finding the optimal learning strategy for an E-learning system. To this end, we first develop a model for students' hierarchical skills in the E-learning system. Based on the hierarchical skill model and the classical cognitive diagnosis model, we further develop a framework to model various proficiency levels of hierarchical skills. The optimal learning strategy on top of the hierarchical structure is found by applying a model-free reinforcement learning method, which does not require information on students' learning transition process. The effectiveness of the proposed framework is demonstrated via numerical experiments.
BibTeX
@misc{li2018optimal,
  author = {Xiao Li and Hanchen Xu and Jinming Zhang and Hua‐Hua Chang},
  title = {{Optimal Hierarchical Learning Path Design with Reinforcement Learning}},
  year = {2018},
  eprint = {1810.05347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.05347},
}
Figure