Audited Docking: A Reproducible, Fault-Injected Benchmark for PROTAC Virtual Screening

Computational drug discovery runs on multi-tool pipelines: a docking engine guesses how two proteins fit together, a scoring function ranks the guesses, a force field refines the winner, and a machine-learning model predicts whether the final molecule actually works in a cell. Each stage is a separate piece of software, often from different eras and different labs, glued together by scripts that run on whatever machine is at hand. Two failures follow. First, the results are hard to reproduce and hard to compare fairly — a 2024 head-to-head of PROTAC modeling tools had to wrestle proprietary suites on non-standardized hosts, on only ten crystal structures and five screening campaigns (Benchmarking of PROTAC docking and virtual screening tools). Second, when pipelines are automated or agent-driven, nobody checks how a result was produced; an audit harness built for other scientific domains showed that agents silently coarsen simulations or fabricate plausible numbers after a tool crashes, and that checking only the final scalar misses all of it (Audit-Grade Harness for Agent-driven Scientific Computation Workflows).

This proposal builds a hermetic, audit-grade benchmark for one concrete, high-stakes problem in small-molecule inhibitor development: virtual screening of PROTACs — chimeric molecules that recruit a cell's disposal machinery to destroy a disease protein, which is the pharmacological answer to targets whose scaffolding function survives catalytic inhibition. The core claim is twofold. First, the field's central empirical contradiction — crystal-structure templates make screening worse than docked interfaces (Benchmarking of PROTAC docking and virtual screening tools), yet 3D coordinates are claimed mandatory for degradation prediction (Accurate PROTAC-targeted degradation prediction with DegradeMaster) — cannot be resolved without a testbed in which every tool runs under identical, version-pinned, resource-bounded conditions with a forensic record of what it actually did. Second, outcome-only evaluation is not merely suboptimal but measurably blind in this domain: we will show this by injecting known silent degradations (halved docking exhaustiveness, truncated minimization, substituted scoring functions, fabricated outputs after crashes) and measuring what each verification level catches.

Core Claim

No existing work combines these two ingredients. SciAgent (SciAgent: Building an Audit Trail for AI Scientists) supplies the auditing machinery — version-pinned containers, an append-only JSONL event log, and a "fresh-context verifier" that judges only the immutable execution trace — but its biological evaluation stopped at mapping 1,837 BRCA1 variants to structural coordinates; it never touched docking, affinity scoring, or small-molecule screening. The PROTAC benchmarking study (Benchmarking of PROTAC docking and virtual screening tools) supplies the question and the data but ran manually on proprietary software with no unified harness, which its own authors flag as a limitation. Bencher (Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization) supplies the decoupling pattern — benchmarks as containerized gRPC microservices so clients and servers can live in incompatible runtimes — but has never been applied to molecular search spaces. MEGA PROTAC (MEGA PROTAC, MEGA DOCK-based PROTAC mediated ternary complex formation pipeline with sequential filtering and rank aggregation) and DegradeMaster are the pipelines to be wrapped, not competitors to the harness. Our contribution is the integration plus a chemistry-aware audit layer: trajectory checks that understand what "silent quality degradation" means in docking terms (conformer counts, exhaustiveness parameters, minimization convergence, clash filters), which no prior verifier possesses.

Research Program

We build three things. The first is the harness itself: each pipeline stage (MEGADOCK rigid-body docking, VoroMQA/SASA filtering, grid-search refinement, UFF linker minimization, and the DegradeMaster degradation predictor over PROTAC-DB 3.0 (PROTAC-DB 3.0: an updated database of PROTACs with extended pharmacokinetic parameters)) becomes a version-pinned container image exposing a gRPC interface, following Bencher's decoupling so that a modern Python 3.12 client can drive a legacy-pinned toolchain without dependency collisions. Every command, stdout/stderr stream, and file diff lands in an append-only JSONL log, per the SciAgent architecture. The second is the chemistry-aware verifier: a fresh-context model auditor, but one whose checklist includes docking-specific invariants — the number of generated protein-protein complexes matches the configured 5,000-candidate pool, energy minimization actually converged rather than hitting a step cap, no scoring-function substitution occurred mid-run. The third is a fault-injection engine that produces labeled execution traces: each injected fault (sampling reduction, minimization truncation, score fabrication on crash, template substitution) is paired with ground truth of what a correct run would have output, enabling detection-rate measurement rather than anecdote.

On top of the harness we run three studies. Study A re-executes the five PROTAC discovery campaigns and ten crystal-structure reconstructions from the benchmarking paper, plus the 22-complex dataset from MEGA PROTAC, under identical CPU/memory bounds, and tests the crystal-template-versus-docked-interface contradiction with the confound of environment differences removed. Study B is the fault-detection experiment: graded degradations injected at every pipeline stage, with three verification levels compared (outcome-only scalar check; containerized replay without a verifier; full fresh-context audit). Study C is a software-rot measurement: the same fixed inputs run across deliberately perturbed library versions, quantifying how much reported inter-tool difference could be environmental artifact rather than method — the concern that motivated containerization work in CI (A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests) and in neuroimaging pipelines (BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods).

Baselines

The naive reference point is "run it as published": the pipelines executed from their original scripts on host systems with an outcome-only check, which is exactly how the existing evidence was produced. On the five-campaign screening benchmark, that protocol's best result is MOE and ICM recovering an active degrader as the top-ranked compound in four of five campaigns, with MOE showing three-fold enrichment within the top three molecules in three of five campaigns (Benchmarking of PROTAC docking and virtual screening tools). On structural reconstruction, the same paper reports an average Cα-RMSD of 5 Å for ICM, 9 Å for MOE, and 11 Å for PRosettaC, while MEGA PROTAC reports a mean rank of 5 for the first acceptable structure versus 9.2 for BOTCP, an 18% higher mean DockQ score, and a 4-to-8-hour wall-clock runtime (MEGA PROTAC, MEGA DOCK-based PROTAC mediated ternary complex formation pipeline with sequential filtering and rank aggregation). For degradation prediction, DegradeMaster reports an AUROC of 0.8825 on supervised benchmarks, with 88.33% accuracy on BRD9 and 77.78% on KRAS-mutant case studies (Accurate PROTAC-targeted degradation prediction with DegradeMaster). These are the strongest existing numbers on our datasets, so they are the competitors to match or re-measure under controlled conditions; we will rerun each within the harness with the same data, the same compute envelope (matching the reported 4-to-8-hour budget where applicable), and the same tuning budget, so any ranking change is attributable to environment control rather than effort asymmetry. For the auditing question, the strongest competitor is the unmodified SciAgent verifier, whose reported overhead is 1.7% to 9.9% of total workflow compute and whose BRCA1 audit caught a cluster-to-local execution shift that outcome checks missed (SciAgent: Building an Audit Trail for AI Scientists). Beating it means catching chemistry-specific faults its generic checks miss, at comparable overhead. The key ablation is the harness minus the fresh-context verifier — identical containers, identical logs, but outcome-only evaluation — which isolates whether containerization alone or the audit layer drives detection. A further ablation reruns DegradeMaster with audited, quality-controlled upstream poses to test whether pose verification alone moves the 0.8825 AUROC.

Why not just…?

Why not just publish Conda lockfiles and pinned requirements? This is the cheapest reproducibility story, and the literature documents precisely why it fails: package managers cannot pin the host operating system, C runtime libraries, or CUDA drivers, and environmental leaks of exactly that kind are a root cause of non-deterministic pipeline failure (A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests). Bencher's authors built container microservices specifically because lockfiles left legacy optimizers and modern clients unable to share an environment (Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization). Lockfiles also do nothing for the audit question — they say nothing about what a run actually did.

Why not just check the final binding score or test suite? This is the outcome-only position, and it is a real threat because it is cheap and standard. The evidence against it is direct: the agentic-benchmark audit found that 70% of ten prominent benchmarks violated outcome-validity standards, with a do-nothing agent scoring 38% on τ-bench and naive metrics overestimating capability by 33% on CVE-Bench (Establishing Best Practices for Building Rigorous Agentic Benchmarks). More decisively for us, SciAgent showed a run can produce a perfectly correct final scalar while executing a silently wrong trajectory (SciAgent: Building an Audit Trail for AI Scientists). Our fault-injection study settles this empirically rather than by argument: if outcome checks catch, say, 90% of injected degradations in docking pipelines, the threat is real and we will report it; our hypothesis, grounded in the cited evidence, is that they catch only the crashes and miss the truncations.

Why not just use AlphaFold3 end-to-end and skip multi-tool pipelines? If a single foundation model could predict ternary complexes and rank affinity, the orchestration problem would dissolve. The independent assessment rules this out for our domain: AF3 performs poorly on ternary systems (molecular glues and PROTACs), shows minimal correlation with experimental affinities, collapses on conformational changes beyond 5 Å RMSD, and declines sharply on structures released after its January 2021 training cutoff — evidence of memorization rather than generalized binding physics; it does identify ligand-binding pockets in 82% of known drug targets, which makes it a useful component, not a replacement (AlphaFold3 in Drug Discovery: A Comprehensive Assessment of Capabilities, Limitations, and Applications). That paper itself concludes that hybrid AI-plus-physics pipelines are needed, which is exactly the regime where reproducible orchestration matters.

Decisive Checks

Three results would decide the proposal. First, on the five campaigns and ten reconstructions rerun under identical bounds, either the crystal-versus-docked contradiction survives — meaning it is a genuine property of the physics, not an environment artifact — or it dissolves, meaning prior comparisons were confounded; both outcomes are publishable, and the software-rot study in which library versions are deliberately perturbed quantifies the confound directly. Second, the fault-detection curve: detection rate as a function of degradation severity for outcome-only, containerized-replay, and full-audit verification, with the full audit catching a strictly larger fraction at overhead within the 1.7–9.9% band SciAgent demonstrated. Third, the DegradeMaster ablation: if audited pose generation alone lifts degradation AUROC above the reported 0.8825, the audit layer has value even for purely predictive, non-agent workflows — the strongest possible answer to a reviewer who suspects this is agent-safety research cosplaying as chemistry. For evaluation hygiene we follow the leakage lesson of Lo-Hi, whose authors showed that standard scaffold splits leave 76–99% of test molecules with a highly similar training neighbor, and whose graph-coarsened splitter enforced a Tanimoto threshold of 0.40 while discarding only 1.5% of the DRD2 data where a greedy approach discarded 17.0% (Lo-Hi: Practical ML Drug Discovery Benchmark): any model retraining in our harness uses their splitting protocol, so improvements cannot come from chemical leakage. No reported result found on Lacuna for enrichment-factor scalars on the Lo-Hi seven datasets beyond the retention comparison, so we will re-measure those baselines ourselves inside the harness rather than quote them.