Ranking Enzyme Variants Where Stability Metrics Disagree

In generative audio and music, a settled lesson is that automated scores can diverge sharply from what actually matters: a generated song can win on Fréchet distance while sounding chaotic, and strict signal-fidelity metrics punish exactly the expressive deviations expert listeners value, as documented for codec-style alignment metrics by ViSQOL v3, for source-separation metrics by Beck et al. in A Unified Neural Architecture for Instrumental Audio Tasks, and for distribution-matching scores by Hung et al. in the FreeSound Loop benchmarking initiative. The fix that emerged is to stop trusting a single scalar and instead learn multi-dimensional reward models from expert preference comparisons, conditioned on context: SongEval showed a model trained directly on human aesthetic ratings reaches a Pearson correlation of 0.912 with listeners where general objective metrics manage only 0.61–0.66, and CMI-RewardBench showed that context flips what counts as a defect — low fidelity is an asset on a lo-fi prompt — so the reward model must be conditioned on intent.

Industrial enzyme engineering has the same structural failure with higher stakes. Enzymes for bioreactors, food-processing streams, and continuous biosensors are ranked by cheap computational proxies — free-energy changes ($\Delta\Delta G$), predicted melting temperatures, docking scores, structural-confidence scores — and these proxies disagree both with each other and with wet-lab outcomes. FireProtDB 2.0 curates 5,465,664 experiments across 2,762 proteins and explicitly reports that variants stabilizing under $\Delta\Delta G$ often register as neutral or destabilizing under $\Delta T_m$ — and then discards these discordant entries as "inconclusive." Worse, Havlásek et al. showed in Decoding Protein Stabilization that a stabilized haloalkane dehalogenase (LinB116) aggregates prematurely because partial unfolding exposes cryptic aggregation-prone regions: 60% of residues exposed during unfolding were inside predicted aggregation-prone regions, so the high-$T_m$ variant is less developable in practice. This is the exact biological twin of the perceptual-objective gap, and the computational-experimental validation literature (Kim et al., Kirjner et al., Poddiakov et al., summarized in The Computational Experimental Validation Gap) has described the Goodhart dynamics but built nothing to fix it.

Core Claim

The discordant entries that FireProtDB 2.0 throws away are not noise; they are the enzyme analog of the audio samples where objective metrics and human listeners disagree, and they are precisely the training signal needed to learn a better evaluator. We propose to build a context-conditioned, multi-head preference model for enzyme variants — the methodological transfer of CMI-RM's two-torch architecture — trained on pairwise comparisons derived from multi-assay disagreement, and to show three things: that it predicts functional outcomes on discordant variants better than any single proxy; that the stability and activity evaluation heads are entangled in a measurable way (the biological counterpart of the $\sim$0.85 Spearman correlation CMI-RM found between its musicality and alignment heads); and that, used as a search reward, it resists the reward hacking that single-proxy rewards invite.

Research Program

The first deliverable is a benchmark, built from FireProtDB 2.0, of pairwise variant comparisons labeled per axis: thermodynamic stability ($\Delta\Delta G$), apparent stability ($\Delta T_m$), and, where measurements exist, activity and solubility. Where two assays on the same variant disagree, that pair becomes a labeled preference on the axis the proxy missed. We oversample the high-temperature regime above 70°C, because the Segment Transformer work documented that mesophilic enzymes dominate public data and degrade all temperature predictors in that band. A large pseudo-labeled split, in the spirit of CMI-Pref-Pseudo's 110,000 pairs, is generated from multi-assay consistency rules rather than an LLM, and a smaller expert-curated split is reserved for aggregation outcomes, taking Havlásek et al.'s cryptic-APR mechanism as the labeling guide.

The second deliverable is the model. Following CMI-RM's design, one tower encodes the variant — sequence plus predicted structure plus segment-level features, borrowing the segment embeddings that let the Segment Transformer reach an RMSE of 24.03 and MAE of 18.09 on its 3,454-entry BRENDA-derived dataset. The second tower encodes the operating context: temperature, pH, substrate, and intended deployment. This is the direct transfer of CMI-RM's finding that prompt context determines defect status; here, a 37°C biosensor makes marginal thermostability acceptable while a 70°C batch process does not, and the same variant should be ranked differently under the two conditions. A joint interaction layer fuses the towers, and separate heads predict per-axis preference scores $s_k(v, c)$ for variant $v$ under context $c$, with an orthogonality penalty to push stability and activity heads apart so their residual correlation can be measured rather than assumed away.

The third deliverable is the search integration and hacking test. The scalar reward fed to a search procedure is the bottleneck combination

$$r(v, c) = \min_k \; s_k(v, c),$$

which a search can only raise by improving every axis at once; a weighted sum, by contrast, can be gamed by saturating one head, which is exactly the failure CMI-RewardBench documented when reward models are placed in optimization loops. We plug $r$ into a ProtInvTree-style reward-guided tree search — ProtInvTree's Monte Carlo tree search over a frozen protein language model reached a self-consistency TM-score of 0.881 on CATH 4.2 versus 0.816 for base ESM-3, making it the strongest available search engine — and measure whether single-proxy-guided search drifts into aggregation-prone, discordant regions where the multi-head reward does not. Robustness of the selected designs across sampling distributions is checked with the stability indicator from SOO-Bench, whose codebase is public.

Baselines

The naive reference point is the standard rational-design recipe: rank candidate mutants by computed $\Delta\Delta G$ with evolutionary-conservation filtering, exactly as FireProt does in FireProt: Energy- and Evolution-Based Computational Design of Thermostable Multiple-Point Mutants. No reported prediction accuracy is found on Lacuna for FireProt's ranking itself, but its failure mode is documented: FireProtDB 2.0 reports systematic $\Delta\Delta G$ versus $\Delta T_m$ discordance, and Havlásek et al. showed the energy-stabilized LinB116 aggregates early. Beating this baseline means ranking discordant pairs correctly more often than $\Delta\Delta G$ ordering does.

The strongest existing predictor is the Segment Transformer from Modeling enzyme temperature stability from sequence segment perspective, which reports an RMSE of 24.03 and MAE of 18.09 on its 3,454-entry dataset and outperforms TemStaPro and DeepET above 70°C. It is the strongest competitor because it already addresses the mesophilic-bias problem with a weighted loss and segment attention. To be fair, we retrain and evaluate it on our identical splits with an equal tuning budget, and compare both on pairwise preference accuracy over discordant pairs and on RMSE restricted to the >70°C band. Beating it means at least matching its RMSE in the hot band while substantially exceeding its discordant-pair accuracy — which it cannot express at all, since it outputs one temperature number with no aggregation or activity axis and no context conditioning.

The strongest search baseline is ProtInvTree itself, with its original self-consistency TM-score and RMSD rewards, reusing the published MCTS procedure so the only changed ingredient is the reward. Beating it means that, at matched search compute and the same number of evaluated candidates, our reward yields top-ranked designs with lower predicted aggregation-prone-region exposure under simulated unfolding and higher predicted activity retention, at comparable structural quality. We additionally run a zero-shot frontier multimodal LLM evaluator on our preference pairs, motivated by CMI-RewardBench's finding that Gemini 3 Pro reached only 65.80% agreement with human experts on music quality versus 78.20% for the specialized CMI-RM; no reported result is found on Lacuna for MLLM evaluation of enzyme variants, so we establish that number ourselves. Finally, two ablations isolate our ingredients: the same two-tower model with the context tower removed, and a single-head variant, to separate the value of conditioning from the value of multi-axis prediction.

Why not just…?

Why not just use FireProt-style $\Delta\Delta G$ calculations? Because the discordance is already measured: FireProtDB 2.0 documents that $\Delta\Delta G$-stabilizing variants frequently fail to raise $T_m$, and Havlátek et al. show the deeper failure — energy filters are blind to partially unfolded intermediate states where cryptic aggregation-prone regions surface, which is why the stabilized LinB116 aggregates before or during unfolding rather than after. Energy minimization also rigidifies scaffolds and can destroy the active-site flexibility catalysis needs, the trade-off Luo et al. attacked in their lactic dehydrogenase work. Our discordant-pair benchmark directly measures how often $\Delta\Delta G$ ordering gets these cases wrong.

Why not just train a better regression model on BRENDA? This is the most serious alternative, and we treat it as such: the Segment Transformer already handles mesophilic bias and wins in the hot band, so a reviewer could argue for scaling that recipe with more hyperthermophile data. But it produces a single context-free temperature number; it cannot say that a variant is acceptable at 55°C and fatal at 70°C, cannot represent aggregation or activity axes, and cannot learn from the discordant measurements FireProtDB 2.0 discards — its training signal is precisely the subset where proxies agree. Our head-to-head experiment on identical splits settles the question empirically, and if an extended Segment Transformer matches our discordant-pair accuracy, that is itself a finding we will report.

Why not just use a frontier multimodal LLM as a zero-shot judge? The music-domain evidence is unfavorable: in CMI-RewardBench, Gemini 3 Pro managed only 65.80% expert agreement, well short of the specialized model's 78.20%, and the entangled fine-grained judgments (does this variant survive a 70°C hold without aggregating?) require biophysical representations — segment attention, structural encodings — that general LLMs bring to the table only through their protein-language-model cousins. We still run this baseline rather than argue from analogy, since no Lacuna-reported number exists for this setting.

Why not just run ProtInvTree with its existing rewards? That is literally our ablation: identical search, swapped reward. ProtInvTree's rewards are static geometric proxies, and the validation-gap literature (Kim et al., Poddiakov et al.) shows geometric similarity does not buy stability or solubility. The decisive measurement is what fraction of top designs from each reward land in the aggregation-prone, discordant region under unfolding simulation.

Decisive Checks

Three checks decide the claim. First, retrospective case studies: the model must rank LinB116 as riskier than wild-type LinB on the aggregation axis despite its $T_m$ gain, and, on the Humicola insolens cutinase candidates from the Segment Transformer paper, must place the A78E variant — which delivered a 1.64-fold activity improvement after heat treatment and extended the 60°C half-life from 6.8 to 29.5 minutes — near the top of its 17 candidates rather than ranking it mid-list on raw thermostability. Second, disentanglement: report the Spearman correlation between stability and activity heads across the benchmark and compare it to CMI-RM's $\sim$0.85; a substantially lower value after the orthogonality penalty is evidence the trade-off is being modeled, not blurred. Third, the hacking test: run matched-compute tree searches under single-proxy rewards and under the bottleneck reward, then subject the top designs from each to the Havlátek-style protocol — simulated unfolding at elevated temperature with Markov-state analysis of which regions expose — and count designs whose exposed residues fall inside predicted aggregation-prone regions. If single-proxy search produces them at a high rate and the multi-head reward does not, the transfer works; if both rates are similar, the perceptual-objective gap in enzymes is not fixable at the reward level, and the benchmark itself will still be the first instrument that can detect the difference.