Weakness Localization Is Not Enough: State-Aware Recursive Knowledge Improvement for Large Language Models

Runze Jiang, Yan Wang, Boran Jin, Bingyu Zhu, Longtao HuangarXiv 2026 · arXiv

TL;DR

Knowing where a Large Language Model (LLM) is failing isn't enough to fix it. This paper introduces Recursive Knowledge Improvement (RKI), a framework that treats knowledge updates as a closed-loop control process, estimating a model's internal "operational state" to select the most effective intervention strategy. By distinguishing between whether a model lacks knowledge, can't retrieve it, or can't apply it, RKI outperforms existing methods by over 18 points on average.

What Problem Does This Paper Address?

Current methods for improving LLM knowledge typically follow a "localize and repair" pipeline: identify a specific topic where the model is weak and fine-tune it on related data. However, the authors argue this suffers from knowledge-state aliasing.

Imagine two models failing the same question. Model A might fail because it never saw the fact (an availability issue). Model B might have the fact in its weights but can't retrieve it reliably (an accessibility issue). A third model might retrieve the fact but fail to reason with it (a usability issue).

If you treat all these failures the same way—say, by just showing the model the raw text again—you might fix Model A but fail to help Model B or C. Furthermore, as you update a model, its state changes. A fix for availability might move the model into an accessibility bottleneck, requiring a different type of training in the next round.

The concept of knowledge-state aliasing: same weakness, different underlying states.

Methods

The authors propose Recursive Knowledge Improvement (RKI), which moves away from one-shot updates toward a sequential, state-aware process.

  1. Graph-Structured Knowledge Units: RKI organizes knowledge into structured units (like relations, chains, and trees). These units serve as a "shared address space" for both diagnostic evaluation and targeted training.
  2. Operational State Estimation: Instead of a binary "pass/fail," RKI estimates a profile across six dimensions: acquisition, recognition, recall, understanding, reasoning, and application.
  3. State-Aware Intervention: Based on the estimated state, RKI selects from three intervention families:
  4. Acquisition-oriented: Factual exposure for missing knowledge.
  5. Recall-oriented: Supervision focused on reliable retrieval.
  6. Use-oriented: Examples requiring reasoning and application.
  7. Recursive Loop: After an update, RKI doesn't stop. It re-evaluates the model, re-estimates the new state, and selects the next best intervention.
Figure 2: Overview of RKI. RKI aligns diagnosis and intervention over graph-structured knowledge units and recursively adapts interventions to the model’s evolving operational knowledge state.

Datasets

The researchers evaluated RKI on two primary benchmarks:

  • HotpotEval: A public benchmark derived from HotpotQA, requiring multi-hop reasoning over general-domain facts.
  • SafeInternalEval: A new, challenging benchmark constructed from enterprise-level safety and compliance documents, designed to test knowledge incorporation in a specialized domain.

Key Insights and Ideas

  • Weakness $\neq$ State: The paper empirically proves that the same failure signal hides different states. For example, among units where the model failed a "recall" task, nearly 38% of the time it could still "recognize" the correct answer in a multiple-choice setting.
  • States are Dynamic: A crucial finding is that states evolve even when the outward error persists. In one experiment, after a first intervention, 57.1% of units that still failed an application task showed a significant shift in their underlying recognition-recall patterns.
  • Stale Information is Dangerous: The authors found that using a "frozen" state estimate (estimating the state once and reusing it for multiple update rounds) actually performed worse than ignoring the state entirely. This highlights the necessity of the recursive, closed-loop approach.
Figure 3: Knowledge-state aliasing under the same localized failure signal. Left: recognition outcomes for recall-failure instances. Right: recognition and recall patterns for understanding-, reasoning-, and application-failure units.

Key Results

RKI demonstrated significant gains over state-of-the-art baselines like Self-QA and GraphGen:

  • Average Improvement: RKI outperformed the strongest baseline by 18.41 points across both benchmarks.
  • Enterprise Performance: On the SafeInternalEval benchmark, RKI achieved a score of 58.92, more than doubling the performance of several baselines (which hovered around 22-28).
  • Ablation Success: Using state-aware selection provided a 5.17 point boost over just using weakness localization alone, proving that the "how to fix" is just as important as the "what to fix."

Limitations and Future Work

The authors acknowledge two main hurdles:

  1. Estimation Accuracy: The "operational state" is a diagnostic abstraction, not a direct measurement of the model's neural weights. Its effectiveness depends heavily on the quality of the diagnostic probes used.
  2. Computational Cost: Because RKI is recursive, it requires multiple rounds of evaluation and fine-tuning, which is more expensive than traditional one-shot knowledge injection.

Future work will focus on developing more efficient recursive processes and exploring more flexible, high-dimensional state representations.

Summary updated

Abstract

Knowledge improvement in large language models is shifting from predefined knowledge incorporation pipelines toward evaluation-guided updating. Existing methods use evaluation to localize model-specific knowledge weaknesses and guide targeted post-training. However, identifying what to improve does not necessarily provide sufficient information for deciding how to improve it. The same localized weakness can be compatible with distinct operational knowledge states: the relevant knowledge may be unavailable, available but difficult to access reliably, or accessible but ineffective for downstream use. We call this ambiguity knowledge-state aliasing. When compatible states favor different interventions, greater diagnostic resolution alone is insufficient for intervention selection. Moreover, model updates can change the underlying knowledge state before the observed weakness disappears, such that an intervention appropriate in one round may no longer address the bottleneck in the next. To address these challenges, we propose Recursive Knowledge Improvement (RKI), which formulates knowledge improvement as a sequential, state-aware intervention process. RKI organizes knowledge into graph-structured units that provide a shared basis for diagnostic evaluation and targeted intervention across rounds. At each round, RKI localizes knowledge weaknesses, estimates the model's operational knowledge state from diagnostic evidence and intervention history, selects a corresponding intervention, and re-evaluates the updated model before determining the next intervention. On HotpotEval and SafeInternalEval, RKI outperforms existing knowledge incorporation methods by 18.41 points on average. Further analyses show that knowledge states provide additional guidance for intervention selection beyond weakness localization and can change across model updates even when the observed weakness persists.
BibTeX
@misc{jiang2026weakness,
  author = {Runze Jiang and Yan Wang and Boran Jin and Bingyu Zhu and Longtao Huang},
  title = {{Weakness Localization Is Not Enough: State-Aware Recursive Knowledge Improvement for Large Language Models}},
  year = {2026},
  eprint = {2610.36811},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36811},
}
Figure