When a regulator or a pharmaceutical company wants to know whether a drug harms the heart, the ground truth increasingly comes from an automated process: a large language model reads the drug's FDA package insert and assigns a cardiotoxicity rating. UniTox showed that a GPT-4o pipeline doing exactly this reaches 93% accuracy against the FDA's DICTrank cardiotoxicity benchmark, far above the 78% of prior automated tools, and agrees with clinicians 87–96% of the time UniTox: Leveraging LLMs to Curate a Unified Dataset of Drug-Induced Toxicity from FDA Labels. That number is now quietly becoming the label of record for downstream machine learning: UniTox's own authors trained a graph neural network on its ratings. But nobody has asked whether the judge earned that number by reading the pharmacology, or by reading the furniture of the document.
This matters because LLM judges are known to take shortcuts. Relabel an identical answer as "NEW (2025)" instead of "OLD (1950)" and GPT-4o flips its verdict 30% of the time on general NLP benchmarks, while never once mentioning the cue in its justification — a Cue Acknowledgment Rate of 0% The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge. Judges also reward verbosity and fabricated citations Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. Drug labels are almost perfectly engineered to trigger these biases: vintage 1970s labels are terse, modern 2020s labels carry sprawling pharmacovigilance tables, and citation density varies with regulatory history, not with cardiac risk. The core claim of this proposal is twofold: first, that shortcut bias measurably corrupts LLM curation of cardiotoxicity labels, concentrated exactly on the boundary cases where the rating matters most; and second, that a judge forced to argue mechanistically — step by step down the causal cascade from ion-channel interaction to arrhythmia, citing the label span behind each step — is both more robust to these shortcuts and more accurate than UniTox's flat-summary judge.
Background in Two Sentences
The hERG channel is a potassium channel in heart muscle cells whose job is to end each heartbeat by letting potassium ions flow out, resetting the cell; drugs that block it (by physically plugging the pore, as nicardipine does at the F656 residue Inhibitory Effect of Nicardipine on hERG Channel, or by suppressing the channel's production and trafficking, as arsenic trioxide does via a MALAT1/calpain-1 degradation cascade LncRNA MALAT1/Calpain-1 Axis in ATO Induced hERG Channel Deficiency) stretch the heart's electrical reset, prolong the QT interval on an ECG, and can trigger the frequently fatal arrhythmia Torsades de Pointes. This gives cardiotoxicity something most toxicity endpoints lack: a well-characterized, ordered causal chain that a judge can be forced to walk, which is precisely the structure that step-level LLM evaluation was built to exploit — AgentEval showed that judging each step of a workflow in a directed acyclic graph (a DAG: nodes are steps, edges are dependencies) raised failure-detection recall from 0.41 to 0.89 and achieved 72% root-cause accuracy, near the 81% human expert ceiling AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking — but only on software and customer-service workflows, never on pharmacology.
Core Claim
The proposal claims that the reliability of LLM-curated cardiotoxicity labels is currently unmeasured in the dimension that matters — robustness to document-level shortcut cues — and that causal structure is the fix. Concretely: (i) perturbing only label vintage, verbosity, and citation density, while holding pharmacological content fixed, will shift a nontrivial fraction of UniTox-style verdicts, concentrated in the "less-concern" and "ambiguous-concern" middle bands of DICTrank where most errors live; (ii) a judge restructured to score each node of the cardiotoxic cascade — exposure evidence, hERG interaction evidence (pore block vs. trafficking disruption), cellular consequence, clinical QT/TdP evidence, and dose–exposure context — with a mandatory verbatim citation from the label for every node, will show a lower Verdict Shift Rate under the same perturbations and higher accuracy on DICTrank; and (iii) on the 82 drugs that Insights into Drug Cardiotoxicity from Biological and Chemical Data: The First Public Classifiers for FDA DICTrank flagged as ambiguous-concern and could not resolve with chemistry alone, the structured judge's step-level citations will let clinicians locate the disagreement, which a flat 297-word summary cannot.
Research Program
Three components. First, a perturbation audit adapted from the Silent Judge methodology to drug labels: for each of the 1,318 DICTrank drugs, construct matched label pairs that differ only in a shortcut cue — the label date stamped "1976" versus "2024", the same adverse-event narrative padded with neutral pharmacovigilance boilerplate versus stripped to essentials, identical claims attributed to "a randomized controlled trial" versus "post-marketing reports" versus no source. Measure the Verdict Shift Rate (VSR: the fraction of pairs where the judge's DICTrank-band rating changes) and Cue Acknowledgment Rate (CAR: whether the justification mentions the cue), directly comparable to the +30% recency VSR and 0% CAR reported for GPT-4o on ELI5 The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge. This transfers a validated measurement instrument to a domain where its stakes are highest — a finding echoed from the clinical side, where physicians already distrust "black-box" AI safety tools Clinical Challenges and AI Opportunities in Decision-Making for Cancer Treatment-Induced Cardiotoxicity and where even frontier medical models score below 65% on clinical safety adjudication LLMEval-Med: A Realworld Clinical Benchmark for Medical LLMs with Physician Validation.
Second, the structured judge. Replace UniTox's two-step "summarize, then rate" pipeline with a DAG-conditioned judge: for each drug, the judge must produce a separate evidentiary verdict at each cascade node, each backed by a verbatim quoted span from the label (or an explicit "no evidence in label" entry), and only then integrate the node verdicts into a DICTrank-band rating. This imports AgentEval's step-level evaluation idea AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking] into toxicology, where it has never been applied, and where it has a natural advantage: the DAG is not an artifact of workflow design but the actual pharmacological mechanism, including the critical distinction between acute pore blockade and chronic trafficking downregulation, which short-incubation assays systematically miss Time Is a Critical Factor When Evaluating Oligonucleotide Therapeutics in hERG Assays and which a flat summary tends to blur. The node structure also preserves what UniTox's ternary rating flattens: dose-dependency and exposure context get their own node rather than being averaged into a single label.
Third, cross-examination. Where the structured judge's rating disagrees with a chemistry-only prediction — for instance, the topological Laplacian hERG models that reached 0.981 accuracy on the Ogura benchmark and flagged 227 DrugBank compounds as hERG blockers Virtual screening of DrugBank database for hERG blockers using topological Laplacian-assisted AI models — the judge's cited label spans make the disagreement inspectable: is the chemical prediction of hERG blockade contradicted by clinical evidence, absent from the label, or confirmed but clinically mitigated? This addresses the gap the DICTrank classifier paper left open, namely reconciling cases where descriptors and clinical labels conflict Insights into Drug Cardiotoxicity from Biological and Chemical Data: The First Public Classifiers for FDA DICTrank.
Proposed Experiments
Experiment 1 runs the perturbation audit on the UniTox pipeline replicated exactly (full-context GPT-4o, ternary then binarized rating, as in UniTox), plus at least one other frontier judge for generality, reporting VSR and CAR per cue type and per DICTrank band. The decisive readout is whether VSR in the ambiguous-concern and less-concern bands exceeds roughly 10% — comparable in kind to the general-domain figures — which would mean the 93% headline rests partly on document furniture. Experiment 2 runs the structured judge on all 1,318 DICTrank drugs and reports accuracy against DICTrank (the metric on which UniTox reports 93%) and VSR under the same perturbations, with clinician agreement measured on a random sample to compare against UniTox's reported 87–96%. Experiment 3 injects known errors into labels (a spliced-in QT-prolongation warning for a no-concern drug; a deleted hERG warning for a most-concern drug) and measures whether the node-level citations localize the corruption, using AgentEval's root-cause-accuracy framing (its reported values: 0.89 failure-detection recall, 72% root-cause accuracy, Cohen's $\kappa = 0.84$) as the reference point for what step-level judging buys. Experiment 4 applies the structured judge to the 82 ambiguous-concern drugs and has two clinicians independently assess whether the cited spans make the adjudication tractable. For judge-vs-clinician agreement we report Cohen's $\kappa$; for robustness we report VSR and CAR; for accuracy we report DICTrank accuracy and AUC-ROC on the binarized task so the tabular baseline is comparable. Compute is one frontier API judge plus ablations — modest by design, since the claim is about evaluation methodology, not model scale.
Baselines
The do-nothing reference point is a majority-class predictor that assigns every DICTrank drug to the largest concern band; no reported result found on Lacuna for this baseline on DICTrank, so we will compute it ourselves, and it calibrates how much of the 93% is real signal versus class imbalance. The strongest existing method on this exact question is UniTox's full-context GPT-4o judge, which reports 93% accuracy against DICTrank (versus 78% for prior automated tools) UniTox; we replicate its pipeline verbatim — same model, same full-context prompting rather than retrieval-augmented fragments (which UniTox showed to be worse), same ternary-then-binary rating — and give our structured judge the identical input labels, the same underlying model, and the same single-pass inference budget, differing only in the prompt structure. Beating it means matching or exceeding 93% DICTrank accuracy while showing strictly lower VSR under the perturbation suite; if accuracy ties but VSR drops substantially, that is the headline result. The strongest non-LLM competitor is the Random Forest on 2D Mordred descriptors, reporting AUCPR of 0.93 and AUC-ROC of 0.84 on DICTrank Insights into Drug Cardiotoxicity from Biological and Chemical Data; we rerun it on the same drugs with its published hyperparameter search budget and compare on the binarized task, noting that it cannot be perturbation-tested in the same way since it never reads the label — that asymmetry is itself part of the argument. The key ablation is the same frontier model, same DAG nodes, but with the verbatim-citation requirement removed (free-text justifications only), and separately a generic non-causal rubric of matching length; these isolate whether the causal structure, the forced citation, or merely any structure at all is doing the work.
Why not just…?
Why not just use UniTox as-is, since it already hits 93%? Because the 93% is measured on unperturbed labels and says nothing about what drives it. The Silent Judge showed judges flip verdicts on recency and provenance cues at +30% and +18% VSR while acknowledging the cue 0% of the time The Silent Judge, and CALM showed near-universal verbosity and authority bias across six models Justice or Prejudice?. Drug labels vary enormously in vintage and verbosity for regulatory reasons orthogonal to cardiac risk, so the threat is real, and Experiment 1 is designed precisely to settle it: if VSR comes back near zero, the honest conclusion is that UniTox is more robust than general-domain judging, and that finding is itself worth publishing. This alternative is a genuine possibility, not a strawman.
Why not just train a better tabular classifier instead of auditing the judge? The Mordred Random Forest is already strong (AUCPR 0.93) Insights into Drug Cardiotoxicity from Biological and Chemical Data, but it operates on a different object: it predicts from chemistry and cannot read, cite, or adjudicate label evidence, which is why its own authors left the ambiguous-concern drugs and the text–chemistry conflicts unresolved. The scoping review also warns that tabular cardiotoxicity models suffer data leakage and lack confirmed true negatives Machine Learning for Predicting Human Drug-Induced Toxicity: A Scoping Review. If a reviewer wants pure prediction, the RF baseline stays in the comparison; but it cannot answer the robustness question this proposal targets, because it never sees the document.
Why not just write a better rubric prompt for the judge? This is the most serious alternative, and the ablation is built to concede or defeat it honestly. A generic rubric of matched length is run as a baseline; if it closes most of the VSR gap, then causal pharmacological structure was unnecessary and the contribution reduces to prompt engineering. Two pieces of evidence argue it will not: rubrics alone do not force the judge to ground verdicts in retrievable evidence, and the Silent Judge's 0% CAR shows judges confabulate content-based justifications regardless of how they are instructed to explain themselves The Silent Judge; and AgentEval's step-level gain over flat evaluation (0.89 versus 0.41 recall) came specifically from evaluating intermediate steps in context, not from better end-to-end prompting AgentEval. But the ablation, not the argument, is what decides it.
Decisive Checks
The proposal succeeds if Experiment 1 shows material VSR on boundary-band drugs, and Experiment 2 shows the structured judge cutting that VSR substantially at equal-or-better DICTrank accuracy with citation-grounded justifications that clinicians can audit (κ comparable to or above UniTox's 87–96% agreement range). It fails informatively if perturbations barely move verdicts — then LLM curation of drug labels is more robust than general-domain judging, and the field should know that too — or if the citation ablation shows generic rubrics do all the work. Either outcome is a publishable, decision-relevant answer about whether the automated labels increasingly underneath cardiac-safety machine learning deserve the trust placed in them.
References
- UniTox: Leveraging LLMs to Curate a Unified Dataset of Drug-Induced Toxicity from FDA Labels
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Inhibitory Effect of Nicardipine on hERG Channel
- LncRNA MALAT1/Calpain-1 Axis in ATO Induced hERG Channel Deficiency
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Insights into Drug Cardiotoxicity from Biological and Chemical Data: The First Public Classifiers for FDA DICTrank
- Clinical Challenges and AI Opportunities in Decision-Making for Cancer Treatment-Induced Cardiotoxicity
- LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
- Time Is a Critical Factor When Evaluating Oligonucleotide Therapeutics in hERG Assays
- Virtual screening of DrugBank database for hERG blockers using topological Laplacian-assisted AI models
- Machine Learning for Predicting Human Drug-Induced Cardiotoxicity: A Scoping Review