When a language model tackles an expert problem, such as a physics question that hinges on one theorem, it often fails in two linked ways. It may not know which rule governs the problem. And when we hand it reference text to help, text that shares the right vocabulary but describes the wrong rule pulls it confidently toward a wrong answer. Methods that pick questions or documents by information gain score them by how much they shrink the model's uncertainty over final answers. We argue this is the wrong target in expert domains. A misleading passage can shrink answer uncertainty just as well as a helpful one, by collapsing belief onto a wrong answer. We propose to measure information gain over a different, checkable variable: which domain principle applies to the problem, and how that principle's variables and conditions map onto it. Questions and passages are then chosen and admitted according to how well they discriminate between rival principles.
Concretely, the system keeps a small set of candidate principles for each problem. For each principle it asks targeted condition checks, like "is the process adiabatic?" or "is the series absolutely convergent?". These checks are answered against the problem statement or against retrieved passages. It asks the check that best splits the candidates, and it lets a passage into the prompt only if the passage moves belief between candidates. Solving then proceeds with the chosen principle and its bindings, using code-based reasoning. The claim is testable in one sentence: when near-miss distractors are present, answer-level entropy reduction stops tracking correctness, while principle-level information gain keeps tracking it.
Where this sits in prior work
Three lines of work supply the method. BED-LLM runs sequential Bayesian experimental design with an LLM. It samples hypotheses, scores candidate questions by expected information gain, and updates belief using the likelihood of an answer given a hypothesis rather than the reverse. It gains 37.4 percentage points over direct prompting on 20 Questions. Uncertainty of Thoughts plans questions by simulated entropy reduction, including medical diagnosis and troubleshooting. We reuse BED-LLM's likelihood-based updating and greedy one-step selection. Both papers, however, work over enumerable targets such as objects, diseases, and faults, with clean yes/no answers and no retrieved text. We move the hypothesis space to formal principles and let passages act as noisy observations.
The closest work on noise is InfoReasoner. It rewards search actions that concentrate sampled answers onto the correct cluster and penalizes searches that spread belief out, with answers grouped by general-purpose entailment models. InfoGain-RAG filters documents whose presence lowers the model's confidence in the gold answer. Both define information over answers and need gold answers during training. Both are evaluated on open-domain QA, where a general entailment model can tell answers apart. In theorem-driven problems, two numerically different outputs from the same wrong theorem look like a confident, low-entropy cluster. We replace answer clusters with principle hypotheses, whose identity TheoremQA labels directly. This makes our signal checkable without trusting answer confidence.
Two findings motivate and constrain the design. TheoremQA reports that pasting theorem definitions into the prompt "did not provide a significant boost." That result is a direct threat to any "just find the right theorem" story. We read it as evidence that the missing piece is applying the rule, meaning its conditions and variable bindings, not naming it. So our hypotheses are principle-plus-binding pairs, and our questions are condition checks. BRIGHT shows that retrieval collapses when documents connect to queries through theorems rather than wording. The best model reaches only 22.5 nDCG@10 on average, and BM25 reaches 14.5. BRIGHT also shows that having an LLM reason before searching helps by up to 12.2 points, which is the simple competitor we must beat.
On the noise side, LDAR finds an inverted-U: accuracy first rises with added context, then falls, and small open models degrade fastest. DAS shows readers become overconfident on answers drawn from irrelevant documents. SKR gates retrieval with a known/unknown classifier. SKR's authors leave specialized domains to future work and find that asking the model directly whether it needs help works worst. All three filter on similarity scores or answer confidence. None of them asks whether a passage separates competing explanations, which is the problem their own findings point to for near-miss noise.
Core Claim
Let $H$ be a finite set of hypotheses $h = (p, b)$, where $p$ is a candidate principle and $b$ a binding of its conditions and variables to the problem. Let $q$ be a condition check with possible answers $a$, answered by the problem text or by a retrieved passage. We select
$$q^* = \arg\max_q \; \Big[ H(h) - \mathbb{E}_{a \sim P(a \mid q)} H(h \mid q, a) \Big],$$
where $H(\cdot)$ is Shannon entropy over the current belief. Beliefs are updated with $P(a \mid q, h)$ as in BED-LLM.
A retrieved passage $d$ is admitted only if its realized principle-level gain $H(h) - H(h \mid d)$ exceeds a threshold. The likelihood $P(d \mid h)$ is estimated by asking whether $d$'s content is consistent with each hypothesis's conditions. This quantity never refers to the final answer. Our central claim is that it therefore resists the failure in which a distractor looks informative by making the model confidently wrong.
Research Program
Building hypotheses. TheoremQA's 350 annotated theorems form the principle library. An LLM drafts two to five applicability conditions and a variable schema for each theorem. We hand-audit a sample and report the error rate. For each problem, the top-$K$ principles, with $K$ around 8, are proposed by sampling plus retrieval over the library. Coverage, meaning how often the gold theorem is among the $K$, is reported separately so that selection errors are not confused with proposal errors.
Condition checks. Candidate checks come from the conditions of the surviving hypotheses. Each check is answered with a small categorical set: yes, no, or not-stated. It is answered first against the problem text, and only if the answer is "not-stated" by issuing a retrieval query built from that condition over the BRIGHT theorem corpora. This is where retrieval noise enters, and where the admission rule acts.
Solving. The top hypothesis and its bindings are passed to a program-of-thought solver, which writes Python code to carry out the computation.
Ablations. We compare the same loop with information measured over answer clusters, as InfoReasoner does, against measurement over principles. We also replace the admission rule with similarity top-$k$.
Baselines
The naive reference is program-of-thought prompting with no retrieval and no questions. On TheoremQA, the file's strongest reported figure is GPT-4 with program-of-thought at 51.0% accuracy (TheoremQA), with open models of that era below 15%. We rerun this baseline on our own model set, one open 8B model and one open 70B-class model, and quote 51.0% only as context. A second reference is Theorem Augmentation, which prepends the gold or top-retrieved theorem. This baseline matters because the paper reports that it gives no significant boost.
The strongest competitor is BRIGHT-style reason-then-retrieve. The LLM writes a chain of reasoning about the problem, the reasoning is used as the retrieval query over the theorem corpora, the top passages are prepended, and program-of-thought solves. BRIGHT reports gains of up to 12.2 nDCG@10 for this kind of query expansion (BRIGHT). We found no reported TheoremQA answer-accuracy result for it on Lacuna. It is the strongest competitor because it already targets the principle implicitly, at a fraction of our cost.
The answer-level information-gain baseline applies InfoReasoner's scoring at inference time. It samples 12 answers, clusters them by entailment, and admits passages and picks queries that reduce cluster entropy. Because there are no gold answers at test time, it uses entropy only. No reported result was found on Lacuna for InfoReasoner or InfoGain-RAG on TheoremQA or BRIGHT, and InfoReasoner's 39.1% seven-benchmark QA average is not comparable here. The key ablation holds everything fixed and changes only the variable over which entropy is computed, answers versus principle-binding pairs. This isolates our core ingredient.
Fairness rules apply across all systems. Every system gets the same models, corpora, and principle library. Each system gets the same budget of LLM calls per problem, matched to our method's mean, with unused budget spent on self-consistency sampling. Thresholds and $K$ are tuned on a shared held-out split. Beating the strongest competitor means higher TheoremQA accuracy at equal call budget on both models, significant under a paired bootstrap. It also means a strictly smaller accuracy drop when near-miss distractors are injected.
Why not just…?
Why not just have the model reason first and retrieve with its reasoning? This is the real threat. BRIGHT shows that reasoning-first queries help retrieval, and a strong model may name the right theorem unaided. Our experiments settle it in two places. The first is the matched-budget comparison above. The second is the distractor condition, where reason-then-retrieve has no mechanism to reject a same-vocabulary passage. If reason-then-retrieve matches us in both, the principle-level loop is not worth its cost, and we will say so.
Why not just sample several candidate theorems and take a majority vote? Voting identifies the most popular principle but never tests its conditions. TheoremQA's failed augmentation suggests the error lies in conditions and bindings, not naming. We run voting at equal budget. We also report principle-identification accuracy against TheoremQA's labels separately from final accuracy. If voting names principles as well as we do but solves worse, the condition checks are doing the work.
Why not just filter passages by answer confidence or answer entropy, as InfoGain-RAG and InfoReasoner do? DAS shows readers become highly confident on answers drawn from irrelevant text. That is exactly the case where confidence-based filtering admits the distractor. The answer-versus-principle ablation tests this directly. We additionally log how often each admission rule accepts an injected distractor.
Why not just hand the model the correct theorem? We include gold-theorem injection as an upper bound for naming. If it helps little, as TheoremQA reports, while our gold-principle-plus-checked-bindings variant helps a lot, that directly confirms the claim that application, not identification, is the bottleneck.
Decisive Experiments
Main accuracy test. We report TheoremQA accuracy for all systems at matched call budgets, for both models, broken down by subject: Math, Physics, EE&CS, and Finance.
Retrieval test. We score nDCG@10 of admitted passages on the BRIGHT TheoremQA and AoPS theorem splits using the xlang-ai/BRIGHT release, against BM25 at 14.5 average and the reason-then-retrieve query.
Distractor stress test. For each problem we inject passages describing theorems from the same subject whose conditions fail on that problem. We then plot the correlation between each method's entropy reduction and correctness. The prediction is that answer-level reduction decouples from correctness and principle-level gain does not. SciAux, which pairs ScienceQA items with helpful, irrelevant, or misleading auxiliary text, provides an independent check with naturally written noise. No reported SciAux numbers were found on Lacuna, so all SciAux baselines are our own runs.
Scale test. Following LDAR's observation that small models tolerate noise worst, we expect the gap to be largest on the 8B model.
Transfer test (secondary). We port the loop to NLPEC, with CMeKG relations as the principle library, against KMQA's reported 61.8% test accuracy (KMQA). This test asks whether principle-level gain addresses the 32% of KMQA errors its authors attribute to retrieval noise.
References
- BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design
- Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in LLMs
- Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward
- InfoGain-RAG: Boosting Retrieval-Augmented Generation through Document Information Gain-based Reranking and Filtering
- TheoremQA: A Theorem-driven Question Answering Dataset
- BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
- Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text