Reinforcement learning with verifiable rewards (RLVR) trains a language model by letting it propose answers and rewarding only those that an automatic checker accepts. It has worked where checkers are exact, such as unit tests, proof checkers and optimization solvers. Materials design seems like a natural next domain, because a physics simulation can check a proposed crystal. The problem is that the affordable simulation is systematically wrong. Standard density functional theory (DFT, the workhorse quantum-mechanical method for computing electrons in solids) uses "semilocal" approximations that underestimate band gaps and sometimes call semiconductors metals. The more accurate "hybrid" functionals, which mix in a fraction of exact exchange, fix much of this but cost roughly one extra power of system size. A policy rewarded by the cheap simulation will therefore learn to exploit the cheap simulation's errors, which is reward hacking in a physics costume.
This note proposes to treat semilocal DFT as a noisy verifier whose errors can be predicted from physics, instance by instance. Hybrid DFT is called sparingly, as an appeal court. It is used both to calibrate a model of when the cheap verdict flips and to adjudicate the cases that model is unsure about. The policy is then trained on the corrected probability that a hybrid calculation would accept its proposal. The claim is that this yields more hybrid-verified materials per hybrid CPU-hour than either rewarding the cheap simulation directly or spending the whole budget on hybrid rewards. The reason is that semilocal DFT errors are structured rather than random: they concentrate in narrow-gap, d-electron and charge-delocalizing chemistries, and cheap diagnostics computed from the semilocal calculation itself reveal them.
Core Claim
The closest RL work on noisy checkers, Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers, models verifier error with false-positive and false-negative rates $\rho_0,\rho_1$ and corrects the policy gradient in the forward direction. It reports that backward inversion explodes because it divides by $(1-\rho_0-\rho_1)$. Its rates are effectively global, and they are estimated through an appeals mechanism with an LLM judge. We keep the forward, division-free spirit but make the rates depend on the proposal $x$ and on the semilocal calculation's own outputs $s(x)$:
$$ q(x) \;=\; P\big(r_{\text{hyb}}(x)=1 \mid \tilde r_{\text{sl}}(x),\, s(x)\big), $$
Here $\tilde r_{\text{sl}}\in\{0,1\}$ is the semilocal verdict (for example, "band gap inside the target window"). $r_{\text{hyb}}$ is the verdict under a screened hybrid (HSE06). $s(x)$ collects physically motivated error indicators: the semilocal gap and band-edge orbital character, the fraction of d/f weight at the edges, and the curvature of total energy under a small fractional charge $\delta$, $E(N+\delta)+E(N-\delta)-2E(N)$. The curvature indicator measures departure from the piecewise linearity that exact theory requires. The bounded quantity $q(x)\in[0,1]$ replaces the binary reward.
The physical premise comes from two sources. Role of the self-interaction error in studying chemisorption on graphene from first-principles traces semilocal failures to the convex, non-linear $E(N)$ behavior that hybrids repair. Addressing the Band Gap Problem with a Machine-Learned Exchange Functional shows that fitting eigenvalues through Janak's theorem, which ties energy derivatives in occupation to orbital energies, is what closes the gap error. So the quantity that predicts the cheap verifier's mistakes can be computed cheaply.
What is new is the combination of three things: a reward correction whose noise model is conditioned on the cheap physics calculation, online refitting of that model from uncertainty-gated hybrid appeals, and a demonstration that this matters for policy learning rather than only for screening.
Research Program
The policy is an instruction-tuned 7B language model, the same scale as the Qwen2.5-7B used by Cai et al. It emits a crystal as a parseable specification: a prototype structure from a fixed library of small cells (at most about ten atoms), plus an element assignment. Restricting to prototype substitution keeps every proposal computable and every hybrid call affordable.
The target task is band-gap-window design. Proposals must be charge-balanced and dynamically sensible, and must have an HSE06 gap in a window such as 1.5–2.5 eV, the range relevant to the photocatalysis setting of A novel metal-free two-dimensional material for photocatalytic water splitting – phosphorus nitride (γ-PN). Following the staged rewards of Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling, a proposal must first pass validity: it parses, is charge-balanced, and its semilocal self-consistent calculation converges. Only then does it earn the property reward $q(x)$. We reuse that paper's staging, but our second stage is a corrected physical verdict rather than solver output.
The target window carries an indifference margin of ±0.1 eV. This matches the inter-code band-gap spread for hybrids reported in Reproducibility of Hybrid Density Functional Calculations for Equation-of-State Properties and Band Gaps. Without it we would be training on differences the reference itself cannot resolve.
The correction model is a small calibrated classifier, a gradient-boosted model with isotonic calibration, fit to $(s(x), \tilde r_{\text{sl}}, r_{\text{hyb}})$. It is seeded with a calibration set of a few hundred prototype compounds computed at both levels. During training, a gate decides for each rollout whether to pay for an HSE06 appeal. It fires when the classifier's predictive uncertainty is high, or when the semilocal gap sits within a learned distance of a window edge. The gate borrows its logic from Calibrated Ensemble Disagreement Gates DFT Calls in Reinforcement Learning for Transition-State Discovery, which gates semilocal DFT calls during geometric stepping using ensemble force disagreement.
Appealed samples receive their exact hybrid verdict as reward and are added to the classifier's training set. This keeps the correction calibrated in the regions the policy is actually moving into, which is the main defense against the policy drifting to places where the correction model is wrong.
The work differs from the two neighboring threads in clear ways. From ALCHEMIST we take the gating idea, but ALCHEMIST's oracle is semilocal DFT on gas-phase molecules and its agent controls geometry steps, whereas our agent designs materials and our oracle is hybrid-level. From Derivative Informed Learning of Exchange-Correlation Functionals and CIDER24X we take the lesson that hybrid behavior can be partly recovered from cheaper computations. Both papers, however, distill a functional by supervised learning. We never modify the functional; we correct the reward.
A constraint shapes the chemistry. Hybrid density functional theory benchmark study on lithium manganese oxides shows that the standard 25% exact-exchange fraction over-corrects open-shell manganese oxides, where 5–9% works better. HSE06 is therefore a questionable "truth" for strongly correlated d-electron systems. The main study restricts elements to s/p main-group and closed-shell d0/d10 cations, where HSE06 is a reasonable reference. Open-shell transition-metal chemistries are reported separately as a stress test.
Baselines
The naive reference point is the untrained base model sampling proposals, with every valid proposal sent straight to HSE06 until the hybrid budget is exhausted. This measures how often an LLM stumbles into the window unaided, and every other method must beat it on verified hits per hybrid CPU-hour.
The strongest competitor, and the one this proposal must beat, is screen-then-verify with no RL. The base model (or a supervised fine-tune on known prototype compounds) samples many candidates. Semilocal DFT with a fitted scissor shift, a constant or linear gap correction calibrated on the same seed set, filters them. The top candidates by corrected gap go to HSE06 under the identical hybrid budget. It is strong because it uses the same cheap physics and the same calibration data, and it spends hybrid calls only on promising candidates. It will receive the same semilocal compute, the same seed calibration set, and a tuned filter threshold chosen on held-out seed compounds. The proposal beats it if it finds at least 1.5 times as many distinct HSE06-verified in-window compounds at equal total compute, with comparable chemical diversity measured by distinct element sets and prototypes.
Four RL baselines use the same policy, optimizer, rollout count and total compute. The first is RL on the raw semilocal verdict, the unhandled-noise case. The second is RL with global-rate forward correction in the style of Cai et al., which isolates whether per-instance, physics-conditioned rates matter; it uses a single $(\rho_0,\rho_1)$ pair re-estimated from the same appeals. The third is RL rewarded only by HSE06: clean but scarce, with the hybrid budget spread over fewer rollouts. The fourth is RL rewarded by a learned property regressor with no physics calculation in the loop. It is a text-to-gap model in the style of Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction, trained on the seed hybrid set plus semilocal labels.
Two ablations isolate the key ingredient. One removes the physics indicators $s(x)$ from the correction model, leaving composition and prototype features only; this turns the correction into an ordinary learned reward model. The other replaces uncertainty-gated appeals with the same number of randomly chosen appeals.
For reference numbers, no reported result was found on Lacuna for RL or screen-then-verify on hybrid-verified band-gap design. The nearest reported figures are indirect. Text to Band Gap reports Llama-3 at MAE 0.248 eV and XGBoost at MAE 0.318 eV on AFLOW gaps, which are not hybrid-calibrated. Cai et al. report forward correction raising Math500 Pass@1 from 41.1% to 43.1% for Qwen2.5-7B. Neither is directly comparable, so all headline comparisons are run within this study.
Why not just…?
Why not just screen with semilocal DFT plus a scissor correction, and skip RL? This is the real threat. If a well-tuned filter over base-model samples finds as many hits, the RL machinery is unjustified. The argument for RL is that the proposal distribution itself should move toward productive regions of chemical space, so that hybrid calls are spent on better candidates over time. A scissor shift also cannot capture the state-dependent flips the correction model targets, such as semilocal metals that are gapped under the hybrid. The screen-then-verify baseline above settles this directly. If it wins or ties, we report that and reposition the correction model as a screening tool.
Why not just reward with hybrid DFT and accept fewer rollouts? A clean reward avoids the whole noise problem. The cost is that each HSE06 call on even small cells is far more expensive than its semilocal counterpart, so the policy sees perhaps an order of magnitude fewer rewards. Models That Prove Their Own Correctness illustrates how sample-hungry verifier-feedback RL is: it needed over a month on one GPU for a 6.3M-parameter arithmetic task. The hybrid-only baseline tests whether scarcity or noise is the binding constraint. A hybrid-only win would itself be a useful finding.
Why not just use a learned property predictor as the reward? It is cheap and differentiable-friendly. But a free-floating regressor is exactly the kind of proxy RL learns to game, and Text to Band Gap reports that such models attend mostly to composition rather than crystal geometry. The composition-only ablation and the regressor-reward baseline measure this. We expect the policy to find proposals the regressor scores highly but HSE06 rejects, and we will report that rate.
Why not just correct with global noise rates as Cai et al. do? This is simpler and already published. It suffices only if semilocal errors are roughly uniform across chemistry. The first decisive check below measures whether they are. If disagreement rates are flat across strata, we drop the per-instance model.
Proposed Experiments
The first check tests the premise, before any RL. On the seed calibration set, stratified by band-edge orbital character and semilocal gap, we measure how often the semilocal and HSE06 window verdicts disagree. We then ask whether the physics indicators predict disagreement better than composition alone, using held-out calibration AUC and expected calibration error. The kill criterion is that if the physics features add no held-out predictive power over composition, the per-instance correction has no basis. We would then fall back to global-rate correction and report the negative result.
The second check tests whether hacking actually occurs. During RL on the raw semilocal reward, we periodically send a random sample of in-window proposals to HSE06. We track the fraction that are false positives, and whether the policy concentrates in strata the calibration study flagged as error-prone. A rising false-positive fraction under a rising training reward is the materials analogue of reward hacking, and this measurement is what motivates the correction.
The main experiment runs all methods at matched total compute, with the hybrid budget fixed at a few thousand small-cell HSE06 calls. It reports the following, each over three seeds with bootstrap intervals:
- distinct HSE06-verified in-window compounds per hybrid CPU-hour;
- false discovery rate among the policy's final proposals;
- diversity across prototypes and element sets;
- calibration drift of the correction model over training.
To guard against the reference's own noise, every claimed hit is recomputed in a second code, following the multi-code protocol of Ji et al. Hits that leave the window under the second code are reported separately and not counted.
A final stress test repeats the pipeline on open-shell manganese and iron oxides. There, per Eckhoff et al., the choice of hybrid reference is itself uncertain. This probes whether the correction degrades gracefully when the "true" verifier is contested.
References
- Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
- Role of the self-interaction error in studying chemisorption on graphene from first-principles
- Addressing the Band Gap Problem with a Machine-Learned Exchange Functional
- A novel metal-free two-dimensional material for photocatalytic water splitting – phosphorus nitride (γ-PN)
- Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling
- Reproducibility of Hybrid Density Functional Calculations for Equation-of-State Properties and Band Gaps
- Calibrated Ensemble Disagreement Gates DFT Calls in Reinforcement Learning for Transition-State Discovery
- Derivative Informed Learning of Exchange-Correlation Functionals
- Hybrid density functional theory benchmark study on lithium manganese oxides
- Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction
- Models That Prove Their Own Correctness