Where Resistance Predictors Break: A Tiered Diagnostic Benchmark for Binding-Site Obstruction

When a model that predicts how a mutation changes a drug's binding strength gets a case wrong, we currently have no way to ask why. Was it blind to a bulky side chain now crowding the pocket? Did it miss a distant residue network that reshapes the pocket from afar? Or did it see all of that and still misjudge the thermodynamics? Every published evaluation of mutation-induced drug resistance collapses these questions into a single number — a correlation or error on binding free energy change ($\Delta\Delta G$) — and so cannot answer any of them. Yet the biophysics literature has already organized resistance mechanisms into what is effectively a natural hierarchy: local steric exclusion (a phenylalanine planted where a serine used to be), relational and conformational interference (a loop propped open, an allosteric chain of residues relaying a distortion across the protein), and full thermodynamic synthesis (a flattened free-energy landscape that lets the drug fall off). Meanwhile, a separate line of work on hierarchical task evaluation has shown, in domains from visual hallucination to tool use, that decomposing a hard task into perception, relation, and composition tiers turns an uninformative accuracy score into a diagnostic that pinpoints the "performance cliff" where models fail. Nobody has connected the two.

This proposal builds that connection as a concrete artifact: a three-tier diagnostic benchmark for binding-site obstruction, layered on top of existing resistance datasets, with tier-specific tasks, tier-specific ground truth, and a cross-tier error-attribution analysis. The core claim is twofold. First, that current resistance predictors — graph-signature methods, feature-based tree ensembles, and structure-prediction foundation models alike — fail non-uniformly across tiers, with a sharp cliff at the relational tier (out-of-pocket and conformational mutations), and that flat metrics have hidden this. Second, that the diagnostic is actionable: adding tier-2 features to a standard predictor should close the out-of-pocket gap specifically, leaving in-pocket performance untouched — a targeted repair that a flat evaluation would never have suggested.

Background in Brief

Drug resistance mutations rarely break a protein; they obstruct it. Three obstruction modes recur across the literature. Direct steric exclusion: the IDH1 p.S280F mutation substitutes a small serine with a bulky phenylalanine that physically displaces the inhibitor ivosidenib while the native substrate keeps binding Assessing acquired resistance to IDH1 inhibitor therapy by full-exon IDH1 sequencing and structural modeling, and gatekeeper mutations in tyrosine kinases do the same to ATP-pocket inhibitors while leaving ATP itself room to bind Protein tyrosine kinase inhibitor resistance in malignant tumors: molecular mechanisms and future perspective. Conformational locking: in ROS1, the G2032R mutation is not a static clash at all — the arginine acts as an internal scaffold that props the phosphate-binding loop open, collapsing the drug-unbinding barrier from about 12.10 to 5.53 kcal/mol and raising the IC50 from 2.1 nM to 570 nM P-loop Conformation Governed Crizotinib Resistance in G2032R-Mutated ROS1 Tyrosine Kinase: Clues from Free Energy Landscape. Distant allosteric transmission: the out-of-pocket ALK mutation I1171S relays resistance through a ten-residue contact chain rather than touching the drug directly Deciphering the Shared and Specific Drug Resistance Mechanisms of Anaplastic Lymphoma Kinase via Binding Free Energy Computation. Even insertions can obstruct: a 27-nucleotide repeat in rplV constricts the ribosomal exit tunnel just enough to block macrolides while peptides still pass Twenty-seven-nucleotide repeat insertion in the rplV gene confers specific resistance to macrolide antibiotics in Staphylococcus aureus.

The evaluation side of the ledger is uniformly flat. MdrDB, the largest resistance dataset (100,537 samples across 240 proteins and 440 drugs), benchmarks ten standard regressors purely on $\Delta\Delta G$ correlation and error A mutation-induced drug resistance database (MdrDB). mCSM-lig predicts affinity change from graph signatures of the structural environment, evaluated as a scalar regression mCSM-lig: quantifying the effects of mutations on protein-small molecule affinity in genetic disease and emergence of drug resistance. AlphaFold3 has been stress-tested across many structural pharmacology scenarios, but as separate exercises rather than a diagnostic ladder AlphaFold3 in Drug Discovery: A Comprehensive Assessment of Capabilities, Limitations, and Applications. Hierarchical evaluation, by contrast, has matured in vision and language: datasets that separate object presence from attribute binding from spatial reasoning PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset, and multi-level benchmarks that isolate the locus of tool-use hallucination ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models. The general argument — that probing only the top tier of processing leaves a diagnostic blind spot From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions — transfers naturally to structural biophysics, but the mapping has never been built.

What Is and Is Not Already Done

Four papers come close, and each stops short in a different way, which is exactly what fixes the design here. mHMG-DTI uses a hierarchical representation (atoms to motifs to molecule) but evaluates flat binary and regression metrics, with no resistance modeling and no failure diagnosis mHMG-DTI: a drug-target interaction prediction framework combining modified Hierarchical Molecular Graphs and improved Convolutional Block Attention Module. Protap benchmarks a horizontal suite of five independent protein applications, including mutation effect prediction as a standalone regression, but never decomposes one capability into ascending tiers General Protein Pretraining or Domain-Specific Designs? Benchmarking Protein Modeling on Realistic Applications. The AlphaFold3 assessment varies scenarios but keeps them decoupled — it reports that AF3 fails on conformational changes beyond 5 Å RMSD, yet cannot say whether that failure traces to misread local geometry or to missing loop dynamics AlphaFold3 in Drug Discovery. And Vedithi et al. compute many physical layers — stability, entropy, ligand affinity, contacts — across 21,394 systematic mutations of the M. leprae RNA polymerase beta subunit, but as parallel bioinformatic predictions, not as a formal diagnostic benchmark for machine learning models Computational saturation mutagenesis to predict structural consequences of systematic mutations in the beta subunit of RNA polymerase in Mycobacterium leprae. The open move is to synthesize these: a tiered evaluation with tier-specific ground truth, applied to standard models, with cross-tier error attribution. No paper in the file does this.

Research Program

The benchmark — call it OB-Tiers — annotates every MdrDB sample with computable tier labels and defines a distinct task at each tier. Tier 1 (perception) asks whether a model can detect the local geometric consequence of a mutation: given wild-type and mutant structures with the ligand posed, classify whether the substituted side chain produces a static clash or occlusion with the ligand, and regress the side-chain volume and solvent-accessible surface area change. Ground truth here is geometric and can be computed exactly from the coordinates MdrDB already ships, using the alpha-shape "knob level" machinery of Ma, Zou, and Yan, which quantifies local pocket convexity from molecular dynamics trajectories and correctly categorized 27 of 30 EGFR mutants (90% accuracy) against clinical response Identifying EGFR mutation-induced drug resistance based on alpha shape model analysis of the dynamics. Tier 2 (relation) asks whether the model correctly interprets how components interact: classify the obstruction mechanism — in-pocket steric, out-of-pocket allosteric, conformational/loop, or insertion/indel — using distance-to-ligand bins, contact-network perturbation, and secondary-structure context. Roughly 4.5% of MdrDB samples are insertions or indels, so this stratum exists at usable scale A mutation-induced drug resistance database (MdrDB). Tier 3 (composition) is the existing end-point task: regress $\Delta\Delta G$, and on a small gold subset, predict the direction and magnitude of unbinding-barrier change, where physics ground truth exists (ROS1's 12.10-to-5.53 kcal/mol collapse P-loop Conformation Governed Crizotinib Resistance in G2032R-Mutated ROS1 Tyrosine Kinase).

Beyond the tier tasks, the program's second ingredient is cross-tier error attribution: condition tier-3 regression error on tier-1 and tier-2 correctness, per model, and report a cliff statistic (the performance drop between adjacent tiers). This is what converts the benchmark from a scoreboard into a diagnostic — the analog of what ToolBeHonest does for tool hallucination, relocated to biophysics. The third ingredient is a targeted repair test: extend the strongest tabular baseline with tier-2 features (contact-network perturbation scores, predicted stability changes of the kind Vedithi et al. compute with FoldX, mCSM, ENCOM and DynaMut Computational saturation mutagenesis to predict structural consequences of systematic mutations in the beta subunit of RNA polymerase in Mycobacterium leprae) and test whether the out-of-pocket and conformational strata improve while the in-pocket stratum is unchanged. A uniform improvement would suggest the features are just generically useful; a stratum-specific improvement is the signature that the diagnostic identified the real failure locus.

The gold mechanism case set is small but decisive: the 30-mutant EGFR clinical cohort Identifying EGFR mutation-induced drug resistance based on alpha shape model analysis of the dynamics; ROS1 G2032R with its metadynamics-derived free-energy landscape P-loop Conformation Governed Crizotinib Resistance in G2032R-Mutated ROS1 Tyrosine Kinase; the four ALK mutations spanning in-pocket, out-of-pocket, scaffold-selective, and cooperative mechanisms Deciphering the Shared and Specific Drug Resistance Mechanisms of Anaplastic Lymphoma Kinase; IDH1 p.S280F Assessing acquired resistance to IDH1 inhibitor therapy; the rplV insertion Twenty-seven-nucleotide repeat insertion in the rplV gene; and the M. leprae set of 40 confirmed rifampin-resistant variants Computational saturation mutagenesis. On these, physics simulations supply tier-2 and tier-3 ground truth at a scale where running them is feasible.

Baselines

The do-nothing reference is a pair of trivial predictors on the tier-3 task: predict $\Delta\Delta G = 0$ for every mutation (the mutation changes nothing), and predict the training-set mean $\Delta\Delta G$. These set the floor; any model worth discussing must beat them, and on out-of-pocket strata it is not obvious a priori that everything does. The strongest existing tabular competitor is the MdrDB ExtraTrees model, which on the independent TKI test set reports a Pearson correlation of 0.607 and an RMSE of 0.656 kcal/mol A mutation-induced drug resistance database (MdrDB); we retrain it from its 146 engineered features on our tier-annotated training splits, with the same feature budget and tuning protocol, and evaluate it per tier and per mechanism stratum. The strongest structural-signature competitor is mCSM-lig, which reports a cross-validated Pearson correlation of 0.628 on Platinum (0.737 after removing 10% outliers) mCSM-lig; it is the key test of Gap 3 because it explicitly truncates its structural environment at 10 Å from the ligand, so out-of-pocket allosteric mutations should fall outside its field of view by construction. On the structure-prediction side, AlphaFold3 serves as the tier-1/tier-2 baseline: it correctly identifies ligand-binding pockets in 82% of chemoproteomics targets but fails on conformational adjustments exceeding 5 Å RMSD and shows minimal correlation between its confidence scores and experimental affinities AlphaFold3 in Drug Discovery, so it should pass static clash detection while failing the ROS1-style induced-fit tier-2 task. Finally, the key ablation isolates the proposal's central ingredient: the MdrDB ExtraTrees model with its original tier-1-dominated features versus the same model plus tier-2 contact-network and stability features, on identical splits. Beating the baselines means, concretely: on tier 3, matching or improving the ExtraTrees RMSE of 0.656 kcal/mol overall while substantially reducing it on the out-of-pocket and conformational strata; on tier 2, mechanism-classification accuracy well above what distance-to-ligand alone achieves; and on the attribution analysis, a demonstrated cliff (a sharp tier-to-tier performance drop for at least one model family) that flat evaluation provably cannot express.

Why not just…?

Why not just stratify MdrDB errors by mutation distance to the ligand? This is the cheapest version of the idea and we run it as our first experiment, precisely so the simpler analysis is directly comparable. It will likely show that out-of-pocket samples have higher error. But stratification alone cannot do two things this proposal does: it cannot separate a perception failure from a composition failure within the same stratum (an out-of-pocket error could come from misreading the local substitution or from missing the long-range relay), and it has no ground truth for mechanism — MdrDB ships coordinates and features, not obstruction-mode labels, so tier-2 labels must be constructed, which is part of the work. If the stratified analysis reproduced every finding the tiered benchmark produces, the benchmark's added value would be genuinely questionable; we state that risk plainly and let experiment 1 decide.

Why not just add molecular-dynamics features and improve the RMSE? This is a real threat to the framing, because the evidence says dynamics helps: time-averaged knob levels from MD trajectories beat static crystal snapshots decisively on the EGFR cohort Identifying EGFR mutation-induced drug resistance based on alpha shape model analysis of the dynamics. But running MD at MdrDB scale — 100,537 samples — is computationally infeasible, so a pure "add dynamics features" strategy caps out at case-study scale, and on its own it optimizes the flat number rather than explaining it. Our repair test (experiment 3) engages this directly: cheap tier-2 features from the Vedithi-style toolkit are added on the full dataset, with MD-derived knob levels reserved for the EGFR subset where the 90% accuracy bar is already set. If cheap features match MD features there, the scalable path wins; if not, that is itself a diagnostic finding about where the tier-2 signal actually lives.

Why not just run the physics simulations instead of benchmarking machine learning? Metadynamics and umbrella sampling produced the correct ROS1 answer P-loop Conformation Governed Crizotinib Resistance in G2032R-Mutated ROS1 Tyrosine Kinase, and physics is unambiguously more accurate per case. It is also unusable at scale: these simulations take hours to days per protein-ligand pair and cannot screen the mutation space a clinician actually faces. The proposal's division of labor is deliberate — physics supplies ground truth on the gold case set and oracles for tier 3, while the benchmark measures how much of that accuracy cheaper models recover and where they lose it.

Why not just use AlphaFold3 and its confidence scores? The comprehensive assessment already answers this: AF3 identifies pockets well (82%) but fails beyond 5 Å RMSD conformational change and its confidence does not rank affinities AlphaFold3 in Drug Discovery. Induced-fit obstruction — the ROS1 mechanism — lives exactly in that failure regime. Including AF3 as a tiered baseline converts this known limitation into a measured tier-2 cliff, which is more informative than the current decoupled report.

Proposed Experiments and Decisive Checks

Experiment 1 measures the cliff. Retrain the MdrDB ExtraTrees and mCSM-lig baselines on tier-annotated splits and evaluate all three tiers. The prediction is a sharp tier-2 cliff for mCSM-lig on out-of-pocket samples (its 10 Å cutoff guarantees it cannot see the ALK-style transmission chain) and for AF3 on the induced-fit stratum. The decisive check is the attribution contrast: tier-3 error conditioned on correct versus incorrect tier-1 output. If models fail tier 3 equally often regardless of tier-1 correctness, the tiers are decoupled and the hierarchy adds little; if error propagates, the diagnostic is real.

Experiment 2 runs the gold case set. On ROS1 G2032R, the question is whether any model predicts the direction of the unbinding-barrier collapse (12.10 to 5.53 kcal/mol) from sequence and static structure alone, or whether all of them, like static docking, misread it as a local solvent-front clash P-loop Conformation Governed Crizotinib Resistance in G2032R-Mutated ROS1 Tyrosine Kinase. On the EGFR cohort, the bar is the 90% knob-level accuracy Identifying EGFR mutation-induced drug resistance; on M. leprae, whether models reproduce the finding that 77.5% of confirmed resistant mutations destabilize the scaffold Computational saturation mutagenesis. No reported result found on Lacuna for any learning-based model on the ROS1, ALK, or IDH1 systems, so these cases are genuinely unbenchmarked.

Experiment 3 is the repair test described above: tier-2 features added to ExtraTrees, evaluated per stratum against the 0.607 correlation and 0.656 kcal/mol RMSE reference points A mutation-induced drug resistance database (MdrDB). A stratum-specific improvement on out-of-pocket and conformational cases with in-pocket performance held constant is the result that would confirm the core claim: the hierarchy does not just describe where models fail, it tells you what to add to fix them.

The deliverables are the annotated benchmark and code, the per-model cliff profiles, and — if the claims hold — the first quantitative evidence that resistance prediction fails at a specific, addressable biophysical layer rather than uniformly. That reframing is the point: flat evaluation has been telling us models are mediocre; the tiered evaluation can tell us where, and therefore what the next model needs to see.

References

  1. Assessing acquired resistance to IDH1 inhibitor therapy by full-exon IDH1 sequencing and structural modeling
  2. Protein tyrosine kinase inhibitor resistance in malignant tumors: molecular mechanisms and future perspective
  3. P-loop Conformation Governed Crizotinib Resistance in G2032R-Mutated ROS1 Tyrosine Kinase: Clues from Free Energy Landscape
  4. Deciphering the Shared and Specific Drug Resistance Mechanisms of Anaplastic Lymphoma Kinase via Binding Free Energy Computation
  5. Twenty-seven-nucleotide repeat insertion in therplVgene confers specific resistance to macrolide antibiotics inStaphylococcus aureus
  6. A mutation-induced drug resistance database (MdrDB)
  7. mCSM-lig: quantifying the effects of mutations on protein-small molecule affinity in genetic disease and emergence of drug resistance
  8. AlphaFold3 in Drug Discovery: A Comprehensive Assessment of Capabilities, Limitations, and Applications
  9. PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset
  10. ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
  11. From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions
  12. mHMG-DTI: A Drug-Target Interaction Prediction Framework Combining Modified Hierarchical Molecular Graphs and Improved Convolutional Block Attention Module
  13. General Protein Pretraining or Domain-Specific Designs? Benchmarking Protein Modeling on Realistic Applications
  14. Computational saturation mutagenesis to predict structural consequences of systematic mutations in the beta subunit of RNA polymerase in Mycobacterium leprae
  15. Identifying EGFR mutation-induced drug resistance based on alpha shape model analysis of the dynamics