Benchmarking LLM Next-Step Recommendations with Knowledge Space Theory

Peter Steiner, Jan HochweberarXiv 2026 · arXiv

TL;DR

This paper introduces a formal benchmarking method using Knowledge Space Theory (KST) to evaluate how accurately Large Language Models (LLMs) recommend the "next best step" for a learner. The study finds that while LLMs are remarkably accurate when provided with an explicit domain structure, they struggle to infer prerequisites from task descriptions alone, with performance varying wildly across different mathematical domains and models.

What Problem Does This Paper Address?

As LLMs are increasingly integrated into digital learning tools, a critical question arises: Can they reliably act as adaptive tutors? A core capability of an intelligent tutoring system is recommending content that fits a learner's current progress—neither too easy (already mastered) nor too hard (missing prerequisites).

Existing benchmarks like MMLU focus on subject-matter knowledge, while others evaluate the "fluency" of pedagogical dialogue. However, a fluent explanation can hide a structural error, such as recommending a task for which the student hasn't learned the basics. This paper addresses the lack of a formal, reproducible way to measure the structural accuracy of LLM recommendations against a grounded curriculum.

Methods

The authors leverage Knowledge Space Theory (KST), a mathematical framework that defines a "knowledge space" as a collection of all possible learning states based on prerequisite relations.

  1. The Outer Fringe: For any given state (what a student knows), KST defines the "outer fringe"—the exact set of all unmastered tasks for which all prerequisites have been met. This serves as the deterministic "ground truth."
  2. Prompt Engineering: The researchers tested four structural-information conditions to see how much "help" the LLM needed:
  3. P0 (Descriptions only): The model must infer prerequisites solely from task descriptions.
  4. P1 (Model reconstruction): The model is asked to first reconstruct the prerequisite graph before making a recommendation.
  5. P2 (Reference graph): The model is explicitly given the prerequisite relations.
  6. P3 (Admissible states): The model is given the entire list of valid knowledge states.
  7. Evaluation Metrics: The primary metric is exact-set accuracy—the model gets a point only if its recommendation set perfectly matches the KST outer fringe. They also used F1 scores and Jaccard similarity to measure partial agreement.

Datasets

The study used four existing mathematics domains related to linear functions (Linear Modelling, Slope, Graph to Equation, and Equation to Graph) from the SaTiM project.

  • Scope: 23 total tasks across 50 unique knowledge states.
  • Models: Six API-accessible models, including GPT-5.6 (Sol and Terra), Claude (Fable 5 and Sonnet 5), and Gemini (3.1 Pro and 3.7 Flash).
  • Scale: A total of 2,400 API calls were analyzed.

Key Insights and Ideas

  • Zero-shot Prerequisite Inference is Hard: LLMs are not "natural" curriculum designers. When relying only on task descriptions (P0), exact-set accuracy hovered around 61%. Asking the model to "think" by reconstructing the graph (P1) provided almost no statistical benefit.
  • Structure is a Force Multiplier: Providing the prerequisite graph (P2) or the full knowledge space (P3) boosted accuracy to over 96%. This suggests that for production EdTech tools, it is far safer to compute the "next steps" deterministically and use the LLM only for explaining those steps.
  • Rankings are Unstable: There is no "one model to rule them all." A model that performed best in the "Slope" domain might rank last in "Linear Modelling." This highlights the need for domain-specific benchmarking rather than relying on general LLM leaderboards.
  • Presentation Matters: Even when the information was semantically identical, changing the order of the task catalogue caused models to disagree with their own previous recommendations in about 15% of cases.

Key Results

The study demonstrated a clear hierarchy of performance based on the information provided in the prompt:

Condition Exact-Set Accuracy F1 Score
P0 (Descriptions only) 61.3% 82.2%
P1 (Reconstruction) 62.8% 81.0%
P2 (Reference Graph) 96.3% 98.1%
P3 (Knowledge Space) 98.2% 99.5%
Table 3: Recommendation performance by structural-information condition

As seen in the results, the largest gain came from providing the reference relation (P2) rather than asking the model to figure it out itself. Furthermore, Figure 2 in the paper (not shown here but described in text) illustrates that domain difficulty varies significantly; "Slope" was the hardest domain for ungrounded models, with accuracy dropping as low as 25.8% in the P1 condition.

Limitations and Future Work

  • Small Domain Size: The study was limited to small domains (max 7 components). It remains to be seen how these models handle massive, complex curricula with hundreds of interdependent nodes.
  • Qualitative Gap: The benchmark measures what was recommended, but not why. The authors plan a follow-up qualitative analysis to examine the justifications LLMs provide for their choices.
  • Deterministic Superiority: The authors point out a bit of an irony: if you already have the prerequisite graph to give to the LLM (P2), you don't actually need the LLM to pick the next step—a simple script can do it perfectly. The real value of the LLM lies in contextualizing and explaining those steps to the student, which requires separate evaluation.

Summary updated

Abstract

Selecting an LLM for next-step recommendation requires benchmarks that reflect the intended educational task and domain. We present a benchmarking method that uses Knowledge Space Theory (KST) as a formal evaluation reference. Given a task catalogue and an existing prerequisite relation, the method derives admissible knowledge states and the complete set of immediate next steps for each state. Recommendations generated by a model under a specified prompt are compared with these reference sets. A sample evaluation study uses 2,400 responses from six models across four mathematics domains, four prompt conditions, and two catalogue orders. Exact-set accuracy was 61.3% with descriptions only, 62.8% when models were asked to reconstruct prerequisites, 96.3% with supplied prerequisite relations, and 98.2% with the complete knowledge space. Performance varied across models and domains, and recommendation sets agreed across the two presentations in 85.25% of pairs. The benchmark supports comparisons of model-prompt combinations against a context-specific reference. Its scores describe structural recommendation accuracy under the evaluated conditions; they do not establish learning benefit or the necessity of using an LLM for selection.
BibTeX
@misc{steiner2026benchmarking,
  author = {Peter Steiner and Jan Hochweber},
  title = {{Benchmarking LLM Next-Step Recommendations with Knowledge Space Theory}},
  year = {2026},
  eprint = {2611.13667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13667},
}
Figure