Evolving Application Notes to Close Domain Knowledge Gaps

When a language model fails an expert question, the cause is often not weak reasoning but missing or unusable knowledge. Evidence from university STEM problems is sharp here. Open-source models score below 15% on TheoremQA, and simply pasting the relevant theorem statement into the prompt does not close the gap (TheoremQA). Having the fact in context is not the same as knowing when and how to apply it. Meanwhile, population-based search, which keeps many candidate solutions alive and lets them share information, has recently been pointed at language models, but at the wrong object. AgentPSO evolves reasoning instructions with a particle swarm. On knowledge-heavy medical benchmarks, its authors report that the swarm raises coverage, meaning at least one agent finds the answer, but not majority-vote accuracy, because better procedures cannot supply absent facts.

This note proposes moving the population search off the prompt and onto the knowledge itself. For each domain concept, such as a theorem or a drug–disease relation, we keep a small population of short "application notes." Each note is written only from a trusted source (the theorem statement, or knowledge-graph triples). It states the fact, the conditions under which it applies, and the typical way it enters a solution. Notes are mutated and recombined by a language model, guided by failures on validation questions. They survive according to how much they improve accuracy on held-out questions that use the same concept. The claim is simple: search over how source knowledge is expressed can turn an ineffective fact into a usable one, where neither prompt evolution nor plain retrieval does.

Core Claim

The TheoremQA failure of theorem augmentation and the AgentPSO coverage-versus-consensus split point to the same missing ingredient. The model needs knowledge it can apply, and the search signal that says what makes knowledge applicable is downstream task success. We claim that a population of source-grounded notes per concept, selected by measured accuracy gain on unseen questions and kept diverse by which questions each note fixes, will beat three things on held-out questions: raw source injection, a single LLM-written note, and prompt-level swarm evolution. We also predict where the gain lands. It should appear on questions whose knowledge exists in the source but is not being applied. It should not appear on questions whose knowledge is absent from the source. This is the falsifiable boundary that separates the proposal from "the method just helps."

Research Program

Each individual follows the dual representation of Evolution of Heuristics. EoH pairs a natural-language idea with code and scores it with an exact simulator. Here the individual is a pair: a note plus a pointer to the source passage it must be faithful to. The fitness signal is accuracy of a fixed reader model on validation questions. We reuse EoH's operator styles, adapted to knowledge. An exploration operator writes a structurally different note from two parents. A merge operator combines what two notes share. A repair operator rewrites a note given the reader's failed trace on a question it should have helped. A simplification operator strips content. The repair operator borrows AgentPSO's self-reflective direction, which inspects traces of peers that succeeded where one failed. In our case the "peer" is a different note variant for the same concept.

The fitness of note $n$ for concept $c$ is

$$F(n) = \mathrm{acc}(V_c \mid n) - \mathrm{acc}(V_c \mid s_c),$$

where $V_c$ is the set of validation questions tagged with concept $c$, $s_c$ is the raw source passage, and $\mathrm{acc}$ is the reader's accuracy when that text is placed in context. This measures what the note adds beyond the fact itself.

Two constraints keep the search honest. First, every mutation must pass an entailment check against $s_c$, judged by a separate model, and notes may not share long n-gram spans with any validation question. Without this, notes would drift into hallucinated rules or memorized answers. Second, selection uses niche preservation in "questions-fixed" space. A note that uniquely rescues a question survives even if its average fitness is lower. This is the analogue of MEoH, which preserves populations through code-structure dissimilarity and Pareto selection. Behavioural niches matter more here because textual variety among notes says little about whether they help different questions.

At test time, the best note per concept is retrieved and read once. Inference cost therefore matches single-pass retrieval, unlike multi-agent debate.

The primary testbed is TheoremQA. Its 800 questions over 350 theorems come with the theorem as an explicit knowledge source, and the Theorem Augmentation baseline is already known to fail. That makes it the cleanest place to ask whether expression of identical knowledge matters. We split each theorem's questions into evolution and evaluation halves with two-fold cross-fitting, and report only theorems with at least two questions. Because questions per theorem are few (roughly 2.3 on average), we first audit the distribution and report how many theorems qualify.

The second testbed is NLPEC, the Chinese pharmacist exam, with CMeKG as the source. Hu et al. attribute 44% of their errors to information missing from both text and graph, and 32% to retrieval noise. This lets us stratify gains by error type.

Baselines

The do-nothing reference is the reader model with Program-of-Thought prompting and no added knowledge. Program-of-Thought prompting has the model write a program that computes the answer. On TheoremQA the file reports GPT-4 at 51.0% accuracy with this prompting, and open-source models below 15% (TheoremQA). We run one open-source reader in the sub-15% regime and one API reader, and we re-measure both rather than trusting old numbers.

The second baseline is raw theorem or triple injection, meaning Theorem Augmentation on TheoremQA and top-k CMeKG triples on NLPEC. It isolates whether any gain comes merely from having the fact.

The strongest knowledge-side competitors differ by benchmark. On NLPEC we use KMQA, which reports 61.8% test accuracy against a 60% pass mark (Hu et al.). We also use RoK, which selects knowledge-graph paths by PageRank with an LLM filter and was itself run on CMeKG. RoK is the most direct structured-retrieval rival, and it is fixed rather than searched. No reported result was found on Lacuna for RoK on NLPEC, so we run it ourselves with the same reader.

The strongest search-side competitor is AgentPSO, which reports 72.08% average accuracy with GPT-5.4-mini across five benchmarks. That figure is not on our benchmarks, and no reported result was found on Lacuna for AgentPSO on TheoremQA or NLPEC. We run it in two forms: as published, and with raw source injection added, so that it has access to the same facts.

For fairness, every searched method gets the same evolution question pool, the same reader, and the same total budget of LLM calls during optimization, which we fix in advance and report. All results are on the held-out half.

Two ablations isolate the key ingredient. The first is single-pass distillation: one LLM call per concept writes an application note from the source, with no search. The second is best-of-N, which samples as many independent notes as the evolutionary run evaluates and keeps the best by validation gain. Best-of-N separates "population operators and niching" from "just sampling more." Further ablations remove failure-conditioned repair, remove niching, and remove the entailment check.

Beating the field means a statistically significant held-out gain over both best-of-N and AgentPSO-with-retrieval at matched budget, on both benchmarks, with the gain concentrated in the predicted error strata.

Why not just…?

Why not just use retrieval-augmented generation over the same source? On TheoremQA this is Theorem Augmentation, which the benchmark authors found does not close the gap. On NLPEC, Hu et al. show that plain-text retrieval leaves large residual error from noise. Raw injection is a baseline here, and the proposal fails if notes do not beat it on held-out questions.

Why not just add retrieval to AgentPSO? This is the most natural combination, and it is a real threat. The difference is the unit of search. A single evolved prompt must work across all concepts. A per-concept note can encode concept-specific applicability conditions. The AgentPSO-with-retrieval baseline at matched budget settles which matters more. If it ties our method, the conclusion is that knowledge-level search adds nothing, and we will report that.

Why not just have a strong model write one good note per concept? This is the threat we take most seriously, since a capable writer may already produce usable explanations. The single-pass and best-of-N ablations test it directly. Evolution must win at an equal call budget, and especially on concepts where the single note's measured gain is near zero.

Why not just fine-tune on the domain corpus? AgentFactory shows that adapter tuning and workflow design are complementary, and Towards Large Reasoning Models for Agriculture reports that fine-tuning Phi-3 on AgThoughts moved AgReason success only from 1% to 5%. Fine-tuning also cannot be applied to API readers. As a secondary comparison, we LoRA-tune the open-source reader on the source passages plus the single-pass notes. LoRA is a low-cost method that trains small added weight matrices instead of the full model.

Proposed Experiments

The decisive check is the held-out TheoremQA comparison: evolved notes against best-of-N and single-pass notes at a matched budget, for both readers. The second check is stratification on NLPEC. Using Hu et al.'s error categories, we predict gains in the retrieval-noise and application strata and none in the missing-knowledge stratum. A gain on missing-knowledge questions would signal leakage or hallucinated facts, and would trigger an audit of the entailment filter.

The third check revisits AgentPSO's observation. We measure the gap between coverage and majority vote across five independent readers given evolved notes, versus AgentPSO's evolved skills. The hypothesis is that grounding narrows the gap rather than only raising coverage.

The fourth check is transfer. Notes evolved with the open-source reader are given to the API reader, and vice versa. Transfer would show that the notes encode knowledge rather than reader-specific quirks. Total optimization calls and per-question inference tokens are reported throughout, so that accuracy gains are never bought with hidden compute.