TL;DR
This paper introduces a formal benchmarking method using Knowledge Space Theory (KST) to evaluate how accurately Large Language Models (LLMs) recommend the "next best step" for a learner. The study finds that while LLMs are remarkably accurate when provided with an explicit domain structure, they struggle to infer prerequisites from task descriptions alone, with performance varying wildly across different mathematical domains and models.
What Problem Does This Paper Address?
As LLMs are increasingly integrated into digital learning tools, a critical question arises: Can they reliably act as adaptive tutors? A core capability of an intelligent tutoring system is recommending content that fits a learner's current progress—neither too easy (already mastered) nor too hard (missing prerequisites).
Existing benchmarks like MMLU focus on subject-matter knowledge, while others evaluate the "fluency" of pedagogical dialogue. However, a fluent explanation can hide a structural error, such as recommending a task for which the student hasn't learned the basics. This paper addresses the lack of a formal, reproducible way to measure the structural accuracy of LLM recommendations against a grounded curriculum.
Methods
The authors leverage Knowledge Space Theory (KST), a mathematical framework that defines a "knowledge space" as a collection of all possible learning states based on prerequisite relations.
- The Outer Fringe: For any given state (what a student knows), KST defines the "outer fringe"—the exact set of all unmastered tasks for which all prerequisites have been met. This serves as the deterministic "ground truth."
- Prompt Engineering: The researchers tested four structural-information conditions to see how much "help" the LLM needed:
- P0 (Descriptions only): The model must infer prerequisites solely from task descriptions.
- P1 (Model reconstruction): The model is asked to first reconstruct the prerequisite graph before making a recommendation.
- P2 (Reference graph): The model is explicitly given the prerequisite relations.
- P3 (Admissible states): The model is given the entire list of valid knowledge states.
- Evaluation Metrics: The primary metric is exact-set accuracy—the model gets a point only if its recommendation set perfectly matches the KST outer fringe. They also used F1 scores and Jaccard similarity to measure partial agreement.
Datasets
The study used four existing mathematics domains related to linear functions (Linear Modelling, Slope, Graph to Equation, and Equation to Graph) from the SaTiM project.
- Scope: 23 total tasks across 50 unique knowledge states.
- Models: Six API-accessible models, including GPT-5.6 (Sol and Terra), Claude (Fable 5 and Sonnet 5), and Gemini (3.1 Pro and 3.7 Flash).
- Scale: A total of 2,400 API calls were analyzed.
Key Insights and Ideas
- Zero-shot Prerequisite Inference is Hard: LLMs are not "natural" curriculum designers. When relying only on task descriptions (P0), exact-set accuracy hovered around 61%. Asking the model to "think" by reconstructing the graph (P1) provided almost no statistical benefit.
- Structure is a Force Multiplier: Providing the prerequisite graph (P2) or the full knowledge space (P3) boosted accuracy to over 96%. This suggests that for production EdTech tools, it is far safer to compute the "next steps" deterministically and use the LLM only for explaining those steps.
- Rankings are Unstable: There is no "one model to rule them all." A model that performed best in the "Slope" domain might rank last in "Linear Modelling." This highlights the need for domain-specific benchmarking rather than relying on general LLM leaderboards.
- Presentation Matters: Even when the information was semantically identical, changing the order of the task catalogue caused models to disagree with their own previous recommendations in about 15% of cases.
Key Results
The study demonstrated a clear hierarchy of performance based on the information provided in the prompt:
| Condition | Exact-Set Accuracy | F1 Score |
|---|---|---|
| P0 (Descriptions only) | 61.3% | 82.2% |
| P1 (Reconstruction) | 62.8% | 81.0% |
| P2 (Reference Graph) | 96.3% | 98.1% |
| P3 (Knowledge Space) | 98.2% | 99.5% |
As seen in the results, the largest gain came from providing the reference relation (P2) rather than asking the model to figure it out itself. Furthermore, Figure 2 in the paper (not shown here but described in text) illustrates that domain difficulty varies significantly; "Slope" was the hardest domain for ungrounded models, with accuracy dropping as low as 25.8% in the P1 condition.
Limitations and Future Work
- Small Domain Size: The study was limited to small domains (max 7 components). It remains to be seen how these models handle massive, complex curricula with hundreds of interdependent nodes.
- Qualitative Gap: The benchmark measures what was recommended, but not why. The authors plan a follow-up qualitative analysis to examine the justifications LLMs provide for their choices.
- Deterministic Superiority: The authors point out a bit of an irony: if you already have the prerequisite graph to give to the LLM (P2), you don't actually need the LLM to pick the next step—a simple script can do it perfectly. The real value of the LLM lies in contextualizing and explaining those steps to the student, which requires separate evaluation.
Summary updated