Shortcut Floors for Per-Residue Protein Quality Assessment

Programs that score the quality of a predicted protein structure increasingly do so residue by residue. They mark which stretches of the chain can be trusted and which are probably wrong. Most residues in a reasonable model are fine, though. That means a scorer can look excellent while doing almost nothing useful. It can predict "good" everywhere, or give every residue the model's overall quality score, or flag loops and chain ends because those are usually worse. This is the accuracy paradox (a metric rewarding a model for exploiting class imbalance instead of learning signal), and residue-level evaluation is shaped to invite it. We propose to measure, for each published residue-quality method, how far it actually rises above a set of explicitly built "shortcut" predictors. The comparison uses metrics whose chance level stays fixed when the fraction of bad residues changes.

The two-sentence version is this. Build a floor of cheating predictors that exploit known regularities in residue error, and score every method by its gain over that floor, computed within each structural model with a constant-baseline classification metric. If current leaderboards reorder, or if the gain over the floor is small, then the field has been measuring imbalance rather than error detection. Only then is it worth testing whether a balanced training objective closes the gap.

Core Claim

The evidence that residue quality predictors play it safe is direct. Benchmarking Deep Networks for Predicting Residue-Specific Quality of Individual Protein Models in CASP11 found that CASP11 predictors concentrated their outputs in the 0–5 Å error band and missed large distortions. That paper also reports its own best residue reliability classifier only as raw accuracy (0.77), with no check on what a trivial predictor would score. Later deep methods treat local quality as regression and report correlations. GraphQA regresses lDDT and CAD-score on residue graphs, VoroCNN regresses CAD-score on Voronoi contact graphs, and Ornate does the same with oriented 3D convolutions. None of them compares against shortcut predictors.

Our claim is specific. A large share of the apparent residue-level skill of these methods is matched by predictors that use no residue-specific structural evidence. The most important of these broadcasts the global model score to every residue. When residues from many models are pooled, that broadcast earns high correlation, because variation between models dominates variation within them. The claim can be falsified: if the deep methods clear the floor by a wide margin under within-model balanced metrics, the paradox concern is unfounded for modern methods. That outcome would also be worth publishing.

Research Program

The first piece is the label definition. Each residue is marked unreliable when its reference-based local error exceeds a threshold. We use the 5 Å cutoff of the CASP11 study for comparability, an lDDT cutoff as a superposition-free alternative, and a sweep around both. We report the prevalence of unreliable residues at every threshold, because prevalence is the variable the paradox lives on.

The second piece is the shortcut floor, a family of predictors that each exploit one regularity. The "all reliable" and "all unreliable" predictors are the degenerate cases. The broadcast predictor assigns the model's predicted global score, or even its true global score, to every residue. A positional predictor uses only distance from the chain termini. A sequence-and-exposure predictor uses predicted secondary structure and solvent accessibility, with no 3D model at all. We also include AlphaFold's own per-residue confidence, pLDDT. PSBench shows this confidence is unreliable for ranking complex models, but it remains the free default practitioners actually use. Each continuous shortcut is fit with a single threshold on the training split, the same treatment the real methods get.

The third piece is the metric. Following Good Classification Measures and How to Find Them, no metric can be monotone, a true distance, and have a constant baseline all at once. We deliberately keep the constant baseline and give up the distance property. We are comparing across targets whose bad-residue fraction ranges from a few percent to a majority, so a fixed chance level matters more than triangle-inequality geometry.

The primary metric is Symmetric Balanced Accuracy (SBA), from the same paper. We report Matthews correlation (MCC) and the product of the two class-wise F1 scores (F1-mult) alongside it. The literature disagrees on MCC. Metrics for Evaluation of Word-level Machine Translation Quality Estimation finds it too lenient toward systems that miss minority errors and prefers F1-mult. Semantic Segmentation of 3D Medical Images with 3D Convolutional Neural Networks treats MCC as the right objective under extreme imbalance. Our cheating predictors settle that disagreement empirically in this domain: the metric we trust is the one that ranks the cheaters lowest.

Two rules apply to all of these metrics. Every metric is computed within each structural model and then macro-averaged over models and over targets. Pooling across models is exactly what lets the broadcast predictor win. And, as the MIMIC replicability study warns for macro-averaging, class-wise scores are computed first and then averaged, so models with no bad residues do not distort the result.

The headline quantity is lift. For a method $m$ and metric $S$, lift is $\Delta_m = S(m) - \max_{s \in \mathcal{F}} S(s)$, where $\mathcal{F}$ is the shortcut floor. It is reported with bootstrap intervals over targets.

The fourth piece is conditional on what the audit finds. If lift is small, we test whether training changes it. GraphQA's architecture is retrained under three regimes, with everything else fixed: its original regression loss, prevalence-reweighted regression, and an added differentiable 1 − MCC term on thresholded outputs. The last follows the voxel-grid formulation of the segmentation paper, carried over to residue graphs.

Baselines

The naive reference points are the shortcut floor itself. The all-reliable predictor reproduces the accuracy paradox by construction. The broadcast predictor is the strongest member of the floor and the one most likely to embarrass pooled correlation. These need no tuning beyond one threshold, chosen on training data. A real method "beats" the floor only if its lift interval excludes zero on SBA and on whichever of MCC or F1-mult survives the cheater test.

The strongest existing residue classifier with a reported number is the stacked-denoising-autoencoder method Wang_deep_1 from the CASP11 benchmarking study, at a residue reliability accuracy of 0.77 on 55 CASP11 targets. That paper reports no balanced metric. We will reimplement it on its stated CASP8–9 training residues and score it under our protocol, so its 0.77 can be set against the all-reliable predictor's accuracy at the same threshold. For modern single-model methods, GraphQA, VoroCNN and Ornate are retrained on GraphQA's CASP9–12 split and tested on its roughly 14,000 CASP13 decoys from 72 targets. Each gets the same hyperparameter search budget, so architecture differences are not confounded with tuning effort. No reported result found on Lacuna for residue-level balanced metrics for any of these methods. GraphQA's own local results are described only qualitatively in the file.

For complexes, the reference is GATE-AFM on PSBench, which reports Spearman correlations of 0.283 for TM-score and 0.322 for DockQ_wave on CASP16. These are global, not residue-level, numbers. We use PSBench's residue lDDT labels to evaluate interface residues as their own minority class. The ablation that isolates our key ingredient is the evaluation itself: every method is scored both with pooled correlation and with within-model SBA, and we count how many pairwise rankings flip.

Why not just…?

Why not just keep continuous lDDT regression and Pearson correlation? Pearson correlation is fine as a training target. As an evaluation, it is the most exposed to the broadcast shortcut when residues are pooled across models, and the CASP11 study already shows regressors gravitating to the safe error band. Our experiment answers this directly. If pooled Pearson ranks the broadcast predictor near the deep methods while within-model SBA separates them, correlation is shown to be measuring the wrong thing. If both agree, the reviewer is right and we will say so.

Why not just report F1 on the bad-residue class? Minority-class F1 ignores true negatives, so it rewards over-flagging. In word-level translation quality estimation, phrase-level F1 on the error class ranked an all-bad system above 16 of 17 trained systems (Metrics for Evaluation of Word-level Machine Translation Quality Estimation). Our all-unreliable and positional predictors test whether the same failure occurs on residues.

Why not just use ROC-AUC, or precision–recall curves? ROC-AUC's false-positive rate is divided by the huge pool of good residues, which hides floods of false alarms (The Failure of Standard Metrics on Imbalanced Data). It also says nothing about the single threshold a user must actually apply. Precision–recall area is a serious alternative. It lacks a constant baseline, since its chance level equals prevalence, but we will report it within each model next to SBA and check whether it orders methods the same way.

Why not just reweight the regression loss instead of adding an MCC objective? This is a real threat to the training component, and it is why reweighted regression is one of the three regimes rather than a straw man. If reweighting matches the MCC term on lift, the conclusion is that any prevalence correction suffices. The audit stands either way.

Proposed Experiments

The decisive first check is the floor on CASP11 and CASP13. We compute every method's lift under SBA, MCC and F1-mult, within each model, at a sweep of thresholds. We expect the broadcast predictor to sit close to the deep methods under pooled Pearson but well below them, or not, under within-model SBA. Either result is informative.

The second check is metric validation. We rank the cheating predictors together with the real methods under each metric. Any metric that places a cheater above a trained method is set aside for this domain, which settles the MCC versus F1-mult disagreement for residue data.

The third check is a transfer to discrete lesions. On the 137-antibody Jain set from Challenges in antibody structure prediction, where models under 1 Å global error contain up to 300 D-amino acids and 240 cis-amide errors, we ask whether methods with positive lift on CASP flag these residues above the floor. If they do not, continuous geometric training does not see chemistry. The fourth check repeats the audit on PSBench interface residues, where the minority class is smallest.

The fifth check, run only if lift is small, is the three-regime retraining comparison. Success means positive lift intervals on CASP13 that also carry over to the antibody set.