Knowledge-Conditioned Reducible Loss: Telling Domain Gaps from Noise in Fine-Tuning Data

When a language model is fine-tuned for an expert domain such as physics derivations, finance formulas or appellate law, the training examples it gets most wrong are a mix of two very different things. Some are wrong because the model does not yet know the theorem or legal rule the example depends on. These are the examples the fine-tuning exists to teach. Others are wrong because the example itself is broken: a sabotaged derivation, a mislabeled case, or scraped text that contradicts itself. Hard example mining is the family of methods that steers training toward high-error examples. It cannot tell these two kinds apart, because both look like "high loss." Noise-robust variants make things worse in exactly this setting. They treat whatever a generic reference model finds surprising as unlearnable, and specialized theorems are surprising to a generic model.

We propose a cheap test that separates the two cases. Each training example comes with a worked solution, and we name the knowledge unit that solution relies on, such as a theorem statement or a statute. We then ask how much the model's loss on the gold solution falls when that one unit is placed in its context, compared with placing an unrelated unit of the same type there. If the right theorem makes the model much more confident in the solution, the example is learnable and it is hard because of missing knowledge, so we prioritize it. If no knowledge unit helps, the example is more likely noise or out of reach, so we down-weight it. The resulting fine-tuning set should close more of the domain gap per gradient step, and it should do so without discarding the specialized material the domain needs most.

Core Claim

Our claim concerns a specific signal: the drop in loss on the gold solution when the specific required knowledge unit is in context, minus the drop from an irrelevant unit of the same type. This signal separates "hard because of a knowledge gap" from "hard because corrupted" more accurately than raw loss, than reducible holdout loss with a generalist reference model, and than in-context approximation with demonstrations retrieved by nearest neighbors. Training on data selected or reweighted by this signal should improve theorem-driven problem solving even when no theorem is supplied at test time.

Concretely, let $x$ be a problem and $y$ its gold solution. Let $k$ be the knowledge unit the solution invokes, and let $k'$ be a randomly drawn unit of the same type, for example another theorem from the same field. Let $\ell_\theta(y \mid c)$ be the current model's mean per-token negative log-likelihood of $y$ given context $c$. The knowledge gain is

$$g(x) = \ell_\theta(y \mid k', x) - \ell_\theta(y \mid k, x).$$

The base hardness is $h(x) = \ell_\theta(y \mid x)$. Examples with high $h$ and high $g$ are prioritized. Examples with high $h$ and near-zero $g$ are down-weighted. Examples with low $h$ are treated as already learned. Subtracting the $k'$ term removes the generic effect of having a longer, on-topic context, so $g$ measures whether this particular piece of knowledge explains this particular solution.

Why this is different from what exists

The method takes its scoring idea from two papers. Reducible holdout loss (Prioritized training on points that are learnable, worth learning, and not yet learned) scores an example by how much higher the current model's loss is than the loss of a reference model trained on clean holdout data. That work also showed that plain high-loss selection under 10% label noise does worse than uniform sampling. Sequence Reducible Holdout Loss for Language Model Pretraining carried this to whole sequences in language model pretraining, using the mean over tokens. Both papers depend on the reference model's loss. When the reference model also lacks the theorem, it flags theorem-dependent sequences as irreducible, and selection discards the content the domain needs. We drop the separate reference model entirely. Our "reference" is the current model given the missing knowledge, so the comparison asks whether the gap is a knowledge gap rather than whether some other model already knows the answer.

The closest work is Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning, which approximates reducible loss by comparing a candidate's loss with and without three nearest-neighbor gold holdout examples in the prompt. The nearest neighbors are chosen by a generic sentence embedding. Its overhead is about 1.5% of training time. On GSM8K with 40% sabotaged derivations, it recovered nearly all of the accuracy lost to noise. We change two things. First, the conditioning context is the knowledge unit the solution depends on, not similar examples. Second, we add a matched irrelevant-unit control. Neither change is cosmetic. BRIGHT shows that generic embeddings collapse on theorem-linked relevance: SFR-Embedding-Mistral scores 59.0 nDCG@10 on MTEB but 18.3 on BRIGHT. Nearest-neighbor demonstrations in theorem domains are therefore likely to share surface topic rather than the underlying principle. We avoid retrieval from the question altogether. At training time we have the gold solution, and naming the theorem a worked solution uses is far easier than finding it from the question. We tag it with an LLM and spot-check the tags by hand.

TheoremQA both motivates and constrains the design. It shows that open models score below 15% on theorem-driven problems while GPT-4 reaches 51% with Program-of-Thoughts prompting. It also shows that putting the theorem in the prompt at inference time "did not provide a significant boost." That might seem to undercut the idea, but it actually shapes it. If prompting with the theorem does not make the model produce correct derivations, the bottleneck is applying the theorem to the problem. Our signal does not ask the model to generate anything. It asks whether the theorem makes the gold derivation more likely under the model. That likelihood can rise even when free generation still fails, and those cases are exactly the ones where training on the derivation should teach the model how to apply the theorem.

Because the signal is computed by scoring a fixed solution rather than sampling, it also avoids the problem documented in Hard or Just Unreached?. That paper found that 10–29% of problems with pass@6 = 0 are solvable by the model and simply were not reached by sampling. Difficulty labels built on pass@k inherit that error.

Other related work plays a smaller role here. Earlier hard-mining methods select on raw loss: online hard example mining in detection (Zhang et al.), loss-sorted mini-batches (Ordered SGD), and confidence-weighted clinical extraction (CME2 Net). The noise-trap literature (Jin et al., Rawat et al.) predicts these will fail under the noise we inject. The iterative hard-negative mining behind the KDD Cup 2024 academic QA winner (LLM-Based Iterative Hard Example Mining with Boosting) works in retrieval ranking and has no way to filter noise. We use it only as motivation that domain mining pays off.

Research Program

We build a scorer, a selection and weighting rule, and a controlled testbed where the truth about each example is known.

The testbed is a pool of theorem-linked training problems with worked solutions. The file names no public training corpus of this kind. We will assemble one from theorem-tagged STEM problems that do not overlap with TheoremQA's 800 test questions, and we will check for overlap by matching on both theorem and problem text. Into this pool we inject three kinds of example in known proportions. Clean examples whose theorem the base model already handles form the already-learned class. Clean examples whose theorem the base model fails on form the knowledge-gap class. Corrupted examples follow the ICA protocol, with sabotaged derivations and swapped answers, and we run them at 10% and 40% noise.

With these ground-truth classes we first measure how well each scoring rule recovers the knowledge-gap class and rejects the corrupted class, using area under the precision–recall curve. We then fine-tune a 7B open model under a fixed token budget. Training uses either top-fraction selection, in the spirit of S-RHOL, or soft reweighting, in the spirit of ICA. The scores are recomputed each epoch, because $g$ should shrink once the model has absorbed a theorem.

The second domain tests whether the idea survives outside mathematics. In AppealCase, the 10,000 matched first- and second-instance civil judgments come with cited legal provisions, which serve naturally as $k$. Every one of the 20 models evaluated there scored below 50% F1 on predicting reversal. Here $g$ is computed on the appellate reasoning text given the cited provision versus a random provision from the same cause of action.

Baselines

The naive reference point is uniform fine-tuning on the full noisy pool with the same token budget. Every method must beat it, and it shows how much the injected noise costs. Next to it we run raw high-loss selection, as in Ordered SGD and online hard example mining, with the same top fraction. The noise-trap literature predicts this will do worse than uniform, and confirming that reproduces the problem the proposal targets.

The reference-model baselines are sequence-level reducible holdout loss as in S-RHOL, run in two variants. In the first, the reference model is trained on a generic clean holdout. Our Gap-3 hypothesis predicts that this variant wrongly discards knowledge-gap examples. In the second, the reference model is trained on a domain holdout of the same size as the knowledge-unit library. This variant tests whether simply giving the reference model domain data is enough.

The strongest competitor is ICA (Holdout-Loss-Based Data Selection via In-Context Learning). It is already a cheap, in-context, noise-robust selector for LLM fine-tuning, and it reports nearly full recovery from 40% sabotaged derivations on GSM8K plus 70–80% GPT-4o win rates against uniform training. It has no reported result on TheoremQA or AppealCase that we could find on Lacuna. We run it with its published settings (three nearest-neighbor holdout demonstrations, all-mpnet-base-v2 retrieval, gradient reweighting). It gets the same gold holdout from which we draw knowledge units, the same model, the same token budget, and the same number of hyperparameter trials for its weighting temperature as we use for ours.

Two ablations isolate the key ingredient. In "ICA with oracle demonstrations," the neighbors are chosen to share the gold theorem tag. This separates the effect of conditioning on knowledge from the effect of escaping bad retrieval. Our method without the $k'$ control measures how much the matched irrelevant unit contributes.

We also run knowledge-augmented supervised fine-tuning, which trains on every example with $k$ prepended. This tests whether selection matters at all once the knowledge is available during training.

Beating ICA means three things. Our method must achieve higher precision–recall separation of knowledge-gap from corrupted examples at both noise levels. It must achieve higher TheoremQA accuracy with no theorem in the prompt, outside the variance across three training seeds. And it must achieve higher AppealCase reversal F1, a task on which no model in that paper exceeded 50%. The comparison is against the ablation with oracle demonstrations, not only against ICA as published.

Why not just…?

Why not just use ICA with better retrieval? This is the most serious threat. If ICA with oracle demonstrations sharing the theorem tag matches our method, then the contribution reduces to "fix the retriever," and we will report it that way. The reason to expect a difference is that a worked example confounds the theorem with a specific solution style and specific numbers. A bare theorem statement, contrasted against a matched irrelevant theorem, measures dependence on the principle alone. The oracle ablation settles this directly.

Why not just train on everything with the theorem in context? This attacks the selection step rather than the scoring step. It does nothing about corrupted examples, which still receive full weight. Also, TheoremQA's null result for adding theorems at inference suggests that seeing the theorem is not the bottleneck. The knowledge-augmented fine-tuning baseline under 40% noise tests both points. We also evaluate it with no theorem at test time, the realistic setting.

Why not just filter by pass@k and keep the problems the model sometimes solves, as RL curricula do? Hard or Just Unreached? shows that pass@6 = 0 mislabels 10–29% of problems as hard. In our testbed, pass@k also cannot distinguish a corrupted gold answer, where the model "fails" by being right, from a genuine knowledge gap. We include pass@k filtering at matched sampling compute and expect it to flag corrupted items as hard. It also costs many generations per item, whereas our signal needs three forward passes.

Why not just use a larger base model? TheoremQA's below-15% result held across sixteen open models, and AppealCase's sub-50% F1 held across twenty models up to GPT-4 and DeepSeek-R1. Scale is therefore not an obvious fix. Our method is also complementary to scale, so we additionally report one larger model trained with uniform data as a reference.

Proposed Experiments

The first decisive check is diagnostic and needs no training. On the injected testbed, we compare the precision–recall separation of $g$ with that of raw loss, reducible holdout loss with generic and domain reference models, and ICA. We break the results down by class. The specific prediction is that reducible holdout loss with a generic reference ranks knowledge-gap examples as low-priority at a rate comparable to corrupted examples, and that $g$ does not. If $g$ fails to beat ICA with oracle demonstrations here, we stop.

The second check is downstream. We fine-tune under each baseline at matched budget. We evaluate TheoremQA accuracy with and without theorem augmentation, AppealCase reversal F1 and provision recommendation, and a general-ability retention check. That last check matters because LLaMA Pro documents general-reasoning loss after domain specialization, and focusing on theorem-heavy examples could make it worse.

The third check probes mechanism. We track $g$ for knowledge-gap examples across epochs. If the signal reflects genuine learning, $g$ should fall toward zero as the theorem is absorbed, and the fall should line up with gains on held-out TheoremQA problems that use the same theorem but not on problems that use different ones.

A positive result on the first two checks with a flat $g$ trajectory would suggest the signal works for reasons other than the one we claim, and we would report it as such.