When a language model fails a university-level physics or finance problem, the standard fixes are to hand it the relevant theorem, retrieve related text, or train it on more problems like the one it failed. TheoremQA reports that the first fix does little: prepending definitions of the needed theorems "did not provide a significant boost." We think the fix points at the wrong node. Human tutors know that a student who cannot apply a theorem usually lacks something upstream of it, such as a definition, a lemma, or a manipulation skill the theorem assumes. Intelligent tutoring systems formalize this intuition with a prerequisite graph, a directed map of which concepts must be understood before which others. They probe the learner and trace a failure back to its deepest missing foundation.
We propose to treat the LLM itself as the learner. For each domain, we build a directed prerequisite graph over concepts. We probe the model with short verifiable questions at every node and fit a mastery model that respects the graph: a concept cannot be mastered unless its prerequisites are. From that fit we locate the root gap, the most upstream concept the model has not mastered. Then we remediate that node rather than the concept where the failure surfaced. The central claim is testable: a failure's diagnosed root prerequisite is a better target than either the failed theorem or the whole set of related concepts. This should hold whether the remediation is a worked example added at test time or synthesized practice problems during reinforcement learning.
Core Claim
Prerequisite-aware recommendation, meaning recommendation that respects "learn A before B" order, has been built for human students on closed curricula. Domain-specific knowledge gaps in LLMs have mostly been attacked with flat injection. The two meet in a question neither side has asked: where in a dependency graph does an LLM's domain failure originate? Does acting on that location matter more than acting on the failed concept itself?
Our hypothesis explains the TheoremQA negative result. Theorem augmentation supplied the target node, but the missing knowledge sits one or more edges upstream. If so, diagnosing and supplying the root prerequisite should recover accuracy that theorem augmentation does not. It should also do this with far fewer context tokens than supplying every ancestor.
Relation to Prior Work
Four papers show that directed structure helps LLM domain reasoning, and they are the closest existing work. CAMA builds the most direct competitor. It learns a mathematical causal graph offline by running the PC causal-discovery algorithm on question–concept co-occurrence. At inference, it extracts a question- and trace-conditioned subgraph and verbalizes its edges as "A is a prerequisite for B" statements, lifting DeepSeek-V3 on AIME 2024 from 39.2% to 50.0% Pass@1 (CAMA). CAMA tells the model about the structure. It never tests what the model actually knows at each node, so it cannot tell a missing lemma from a present one. Our proposal adds exactly that measurement step and targets a single node.
DECOMP builds a concept dependency graph with GPT-4o and uses it only to order an offline curriculum. SwS targets the model's weak concepts during reinforcement learning, but it recombines them by co-occurrence and embedding similarity rather than direction. It reports an average +10.0% over baseline RL on Qwen2.5-7B across eight math benchmarks. StructTuning injects hierarchical taxonomy paths. A taxonomy captures containment (a chapter holds its sections) but not the cross-branch prerequisite chains we need. All four motivate the proposal by showing that structure helps. None localizes a gap inside a model.
The educational literature supplies the method. RadarMath traces student failures on Cross Multiplication back to Factorization Definition through a knowledge graph. That is the diagnostic behavior we want, but for a model instead of a child. PSI-KT models learner mastery constrained by a prerequisite graph. CSG generates cognitive structures from interaction logs, reaching Jaccard 0.79 against expert annotations. We borrow their core constraint, that mastery propagates along prerequisite edges. We drop PSI-KT's forgetting dynamics and CSG's diffusion generator, because a frozen model at test time does not forget and does not need a generative structure model.
Two constraints shape the design. Behavioral prerequisite mining, as in DGCPL (F1 0.8557 on the University Course Dataset) and Rank Aggregation for Course Sequence Discovery, needs dense learner logs that do not exist for new domains. We sidestep this: the model's own probe responses serve as the interaction log. Link-based discovery with Reference Distance, the asymmetric Wikipedia-link measure of Measuring Prerequisite Relations Among Concepts, degrades on sparse pages and needs arbitrary thresholds. We therefore use it only as a directional check on LLM-proposed edges, not as the sole source.
Research Program
We work in two domains. The first is TheoremQA's 350 theorems across math, physics, EE&CS and finance, which is the main testbed because it has the documented theorem-augmentation failure. The second is pharmacology via NLPEC with the CMeKG medical knowledge graph, which tests whether the method survives a "fuzzy" domain that DECOMP explicitly leaves open.
Graph construction. For each theorem, a strong model proposes its constituent concepts and their prerequisites, recursively to depth three or four. Each proposed edge is kept if it passes two checks: RefD asymmetry on Wikipedia agrees with its direction when both pages exist, and an independent model agrees under the reversed question. Domain annotators audit a random 10% of edges to report edge precision.
Probing. For every concept node, we generate 8–12 short probe questions with numeric or short answers. Answers are checked by program execution where possible and otherwise by a generative verifier in the style of General-Reasoner, whose verifier reached 78.7% human agreement. We report probe noise separately.
Diagnosis. We fit a graph-constrained mastery model adapted from DINA, a standard cognitive-diagnosis model whose parameters are a probability of "slipping" when a skill is known and of "guessing" when it is not. Let $m_c \in \{0,1\}$ be whether the model intrinsically knows concept $c$. Its effective mastery is then
$$e_c = m_c \prod_{a \in \mathrm{pa}(c)} e_a, \qquad P(\text{probe of } c \text{ correct}) = (1-s_c)^{e_c}\, g_c^{\,1-e_c},$$
where $\mathrm{pa}(c)$ is the set of $c$'s parent prerequisites, $s_c$ is the slip rate and $g_c$ the guess rate. For a failed target problem, the root gap is the ancestor node with the highest posterior probability of "own mastery absent, all parents mastered." Inference is exact on the small ancestor subgraphs involved.
Remediation. At test time, the root node receives a short retrieved or generated worked example, and the target problem is re-attempted. The training arm is secondary. It modifies SwS so that synthesized practice problems start at the root gap and walk down the path to the failed concept, keeping SwS's 25–75% difficulty filter.
Baselines
The naive reference point is the model answering with Program-of-Thought prompting and no added knowledge, run on identical questions and sampling settings. For the frontier ceiling we quote GPT-4 at 51.0% accuracy on TheoremQA with Program-of-Thought (TheoremQA). We will re-measure open models ourselves, since the only open-model figure reported there is "below 15%" for LLaMA and Alpaca.
Theorem augmentation is the baseline our hypothesis is designed to beat. It prepends the target theorem's definition. The paper reports no significant gain, and no exact number for it was found on Lacuna.
The strongest existing structured method is CAMA-style verbalized subgraph prompting over the same prerequisite graph we build. It uses the same subgraph size, token budget and backbone models (Qwen2.5-7B and Qwen3-32B, both used in the cited papers). This isolates diagnosis from graph quality. No reported result was found on Lacuna for CAMA on TheoremQA; its reference numbers are the AIME 2024 gains above.
Two ablations isolate the key ingredient. Ancestor dump gives worked examples for all ancestors, which is diagnosis-free and more expensive. Random-ancestor remediation gives the same token budget but targets a random ancestor rather than the diagnosed one.
For the training arm, the baseline is SwS itself with an equal synthesis budget and equal RL steps. It has a reported +10.0% average on Qwen2.5-7B math and no reported TheoremQA result on Lacuna.
We count the proposal as beating these baselines if root-gap remediation exceeds verbalized-subgraph prompting and theorem augmentation by at least 5 absolute points on failed-then-retried TheoremQA items. It must also match the ancestor dump within 2 points at under a third of its context tokens, with 95% bootstrap intervals over questions.
Why not just…?
Why not just ask the model what it doesn't know? Self-diagnosis costs one prompt, and it is the most serious threat. If models can name their own missing prerequisite, the probing apparatus is unnecessary. We include a self-diagnosis arm, in which the model names the concept it is missing and we remediate that concept. We compare its chosen node against the probe-diagnosed node and against an oracle node found by exhaustively remediating each ancestor on a 200-item subset. If self-diagnosis matches the oracle as often as probing does, we will report that and reframe the contribution around the graph alone.
Why not just dump every prerequisite into the context? This removes the need to diagnose anything. It is exactly our ancestor-dump ablation. We expect it to be competitive in accuracy but several times costlier. TheoremQA's flat-augmentation result and ToG-2's finding that standard retrieval scored 0% on ToG-FinQA (Think-on-Graph 2.0) suggest that undifferentiated context can also distract. If the dump wins outright at acceptable cost, the practical claim weakens, but the diagnostic claim is still testable through the random-ancestor arm.
Why not just use similarity-based weakness targeting like SwS? SwS already works, but it cannot distinguish a concept that fails because of itself from one that fails because of its parent. Its co-occurrence recombination would keep drilling the surface concept. The CAMA ablation reports that dropping edge direction hurts, and RefD outperformed symmetric features. Both are indirect evidence. Our head-to-head at equal synthesis budget is the direct test.
Decisive Checks
Three checks settle the claim. First, the diagnosis must be predictive. The graph-constrained mastery model should predict correctness on held-out probes and target problems with higher AUC than per-concept accuracy that ignores the graph, as in the knowledge-tracing evaluations of PSI-KT and CSG. If it does not, the graph adds nothing to measurement.
Second, the diagnosis must be causal. On the oracle subset, remediating the diagnosed root must fix the target failure more often than remediating the target theorem or a random ancestor. We also report how often the oracle root coincides with the diagnosed one.
Third, the effect must transfer to NLPEC. There the prerequisite edges come from CMeKG relations filtered by our direction checks, and the reference point is KMQA's 61.8% accuracy (Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text). KMQA's error analysis attributes 44% of failures to missing information, which is exactly the category root-gap remediation should shrink. A null result in pharmacology alongside a win in TheoremQA would itself be informative: it would indicate that prerequisite localization works only where dependencies are crisp. That is the open question DECOMP leaves unanswered.
References
- TheoremQA: A Theorem-driven Question Answering Dataset
- CAMA: Enhancing Mathematical Reasoning in Large Language Models with Causal Knowledge
- Learning to Solve Complex Problems via Dataset Decomposition
- SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
- Structure-aware Domain Knowledge Injection for Large Language Models
- RadarMath: An Intelligent Tutoring System for Math Education
- Predictive, scalable and interpretable knowledge tracing on structured domains
- CSG: Cognitive Structure Generation for Intelligent Education
- DGCPL: Dual Graph Distillation for Concept Prerequisite Relation Learning
- Rank Aggregation for Course Sequence Discovery
- Measuring Prerequisite Relations Among Concepts
- General-Reasoner: Advancing LLM Reasoning Across All Domains
- Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text