When a cell turns cancerous, infected, or injured, thousands of genes shift their expression levels together, forming a coordinate pattern — a transcriptomic signature. A long line of work in drug repurposing treats this signature as the disease itself: find a compound whose own effect on gene expression is the mirror image of the disease pattern — pushing down the genes the disease pushes up, and vice versa — and you may have a therapy. The catch, documented empirically, is that mirrors lie. Compounds frequently produce large, genome-wide transcriptional reversals that cause no measurable effect on cell survival or growth, because cells absorb the perturbation through adaptive pathways that statistical matching scores cannot see. Meanwhile, a separate line of work has shown that large language models can act as automated judges — evaluating open-ended outputs with roughly the reliability of a human expert panel — but these judges have only ever been calibrated on conversation, code, and essay-like tasks.
This proposal builds the missing bridge: a benchmark and a structured judging protocol that tests whether an LLM judge, given a candidate signature reversal, can adjudicate between a mechanistically coherent rescue and a generic stress response. The core deliverable is not "GPT-4 ranks drugs" — it is an auditable answer to a precise question: does semantic adjudication of reversal hypotheses carry predictive information about phenotypic outcome that existing statistical scores do not, and under what judging protocols?
Core Claim
The claim is that the gap between molecular reversal and phenotypic rescue — the central documented failure mode of signature-reversal methods — is partly a semantic gap that LLM judges can close, provided they are given a structured adjudication protocol rather than a generic prompt. Existing reversal scores measure how well a compound's transcriptional footprint anti-correlates with a disease signature, but they cannot ask whether the flipped genes are mechanistic drivers or passenger stress genes, whether the compound's known targets plausibly connect to the disease pathway, or whether the reversal is cell-type specific. Chen et al. showed that the summarized Reverse Gene Expression Score (sRGES) correlates with preclinical drug potency, but a correlation is not a mechanism. Fallahi-Sichani et al. showed that strong transcriptomic responses routinely occur with zero growth inhibition — the exact cases where statistical matching produces false positives. My claim is that a reference-guided, step-verifying judge can recover signal on precisely these discordant cases, and that this can be measured against hard ground truth rather than against judge self-reports.
Research Program
Three components. First, a reversal adjudication benchmark assembled from existing public data. The anchor is the Fallahi-Sichani dataset of roughly 600 drug–cell-line pairs across about 8,000 experimental conditions in six breast cancer cell lines plus MCF 10A, where both transcriptional response (LINCS L1000 landmark genes) and normalized Growth Rate inhibition were measured. This gives a labeled discordance set: compounds with large reversal footprints split into those that do and do not actually stop growth. A second tier reuses the Chen et al. evaluation setting — TCGA signatures from 7,514 samples across 14 cancer types, 66,612 LINCS profiles covering 12,442 compounds, and ChEMBL $IC_{50}$ measurements — for additive-validity testing against potency ground truth. A third tier uses the known drug–disease pairs behind PhenoPLIER's evaluation for ranking comparisons.
Second, the judging protocol. Each candidate is rendered as a structured adjudication card: the disease signature's top differentially expressed genes, the compound's LINCS reversal footprint, its known targets from ChEMBL, and the cell-line context. The judge's task is decomposed into verifiable steps following the chain-of-thought and step-level evaluation evidence from MT-Bench and AgentEval: verify the claimed mechanistic chain (target → pathway → cellular process → phenotype) one hop at a time; identify whether the reversed genes are dominated by generic stress or cell-cycle modules versus specific drivers; assess context specificity; then output a score, with an explicit abstain option following Judge, Retrieve, or Abstain when evidence is insufficient. Pairwise comparisons are run in both orderings to cancel position bias, and multi-model committees reduce single-judge variance, both documented mitigations from Zheng et al. and the LLM-as-a-Judge survey.
Third, the combination test. The judge is not proposed as a replacement for statistical scores but as an orthogonal filter. Candidates are ranked by a composite
$$S(c,d) \;=\; z\big(\mathrm{sRGES}(c,d)\big) \;+\; \lambda\, z\big(J(c,d)\big),$$
where $z(\cdot)$ z-scores each component across the candidate pool for disease $d$, $J$ is the judge's score for compound $c$, and $\lambda$ is fit on a validation split. The scientific question is whether $J$ adds predictive information about phenotype and potency beyond sRGES — and, symmetrically, whether sRGES adds information beyond $J$.
This differs from all adjacent systems. LLaDR uses an LLM only as an offline text generator whose embeddings anchor a knowledge-graph model; it never touches expression signatures and never judges anything. TxGNN generates multi-hop mechanistic paths that improved clinician accuracy by 46% in a 12-expert panel — strong evidence that mechanistic rationale matters to adoption — but its paths were evaluated by hand and it is uncoupled from transcriptomics entirely. CURE checks whether generated molecules induce a target signature using PRnet, a regression proxy that offers no explanation and can hallucinate off-distribution; GxVAEs evaluates only chemical validity and structural similarity; PDGrapher and the Large Perturbation Model rank against ground-truth perturbation sets or retrospective EHR records with no semantic critique. None deploys a judge, and none targets the reversal–phenotype discordance directly.
Baselines
The naive reference point is ranking by sRGES alone, the method of Chen et al., which is deterministic, cheap, and the field's reference score; on the same TCGA/LINCS/ChEMBL inputs it will be recomputed so the comparison shares identical candidate pools. That paper reports a Spearman's rho of 0.61 ($P = 1.34 \times 10^{-6}$) between sRGES and drug $IC_{50}$ in liver hepatocellular carcinoma, which is the number the composite score must beat to demonstrate added value. Beating it means a statistically significant improvement in Spearman correlation with $IC_{50}$ in LIHC and at least two additional cancer types, plus materially better-than-chance discrimination on the discordance set where sRGES by construction has no signal; no reported result found on Lacuna for sRGES-based discrimination of growth inhibition on the Fallahi-Sichani discordance set, so we will establish that reference ourselves.
The strongest published competitor is PhenoPLIER, which projects genetic associations through gene-expression modules into 987 latent variables and thereby filters passenger-gene noise; it reported an AUC of 0.632 on known drug–disease pairs versus 0.566 for single-gene reversal matching, making it the best validated reversal-ranking method available. It will be run on the same drug–disease pairs with its published latent space, under the same candidate pools and evaluation splits; beating it means a higher AUC on those pairs, and the more informative test is whether the judge re-scoring PhenoPLIER's top-k hits improves its precision at small k, since a judge is far too expensive to apply to every candidate.
The most dangerous baseline is a supervised phenotypic predictor: a gradient-boosted tree or small multilayer perceptron mapping the landmark signature and compound features directly to Growth Rate inhibition, trained on the Fallahi-Sichani data. It gets the same splits, the same features, and the same tuning budget as the judge's validation procedure. Within-distribution it may well win, and I treat it as the null hypothesis for the entire proposal; the decisive comparison is cross-context transfer (below).
Finally, the key-ingredient ablation is a naive zero-shot judge: the identical LLM, the identical candidate cards, but a single unstructured prompt — "is this compound likely to reverse this disease?" — with no step decomposition, no retrieval anchoring, no permutation, and no abstention. This isolates whether the protocol's structure, rather than the model's raw knowledge, does the work.
Why not just…?
Why not just use sRGES or cosine matching and skip the LLM entirely? These metrics are reproducible and nearly free, but their blindness is not hypothetical: Fallahi-Sichani et al. demonstrated that profound transcriptional changes frequently co-occur with no cytostasis or cytotoxicity, and that kinase inhibitors elicit radically cell-type-specific signatures even when phenotypic outcomes are identical. A rank statistic over a flattened gene list cannot represent "these flipped genes are generic stress passengers," and the LIHC correlation of 0.61, while real, leaves most potency variance unexplained. The proposal's Experiment 1 measures exactly the cases where this failure bites.
Why not just prompt a frontier model zero-shot? This is a genuine threat, and the honest answer is that it might work — or it might produce articulate nonsense. Articulate Intuition or Genuine Analysis? documents LLM judges writing plausible peer-review prose without genuine verification of technical claims, and the survey reports judge performance collapsing by over 40% under pathological reference setups, while shortcut-bias work shows judges seizing on formatting cues. The naive-judge ablation is therefore not a strawman but a first-class experiment: if the unstructured prompt matches the full protocol on ground-truth prediction, I will report that the structure is unnecessary and the field should use the cheap version.
Why not just train a supervised model to predict which reversals cause phenotypic effects? On in-distribution data this may outperform the judge — supervised models exploit local statistics that semantic reasoning cannot. The problem is labeled-data scarcity and context plasticity: the discordance labels come from roughly 600 drug–cell pairs in six breast cancer lines and MCF 10A, while the same paper shows signaling inhibitors produce extreme cell-type-specific responses. The decisive experiment is transfer: train the supervised model on the breast-cancer set and evaluate on held-out contexts (e.g., liver cancer lines from the Chen et al. validation), comparing against the judge, which carries no training-set context bias. If the supervised model transfers as well as the judge, the judge's remaining value is explanation and abstention — a weaker claim, and I would say so.
Why not just use human experts? Humans remain the gold standard for final arbitration, and nothing here replaces that. But Zheng et al. measured inter-expert agreement at only about 80–85% on complex evaluations, and expert panels cannot screen thousands of candidates — the same bottleneck that forced TxGNN to rely on just 12 clinicians. The realistic goal is a judge that matches the 80–85% human-ceiling on a calibrated adjudication subset while running at scale, with humans arbitrating disagreements and the abstention set.
Proposed Experiments
Experiment 1 (discordance discrimination) is the centerpiece: on the Fallahi-Sichani pairs, predict which large-reversal compounds actually inhibit growth, measured by AUROC and AUPRC, comparing the structured judge, the naive judge, the supervised predictor (in-distribution and transfer), and sRGES as a near-null reference. Experiment 2 (additive validity) tests the composite score against $IC_{50}$ in the Chen et al. setting, with the 0.61 LIHC Spearman rho as the number to beat, and against PhenoPLIER's 0.632 AUC on known drug–disease pairs. Experiment 3 (judge reliability) calibrates the judge against a blinded human expert panel on roughly 200 adjudications, reporting agreement against the 80–85% inter-human ceiling, and runs bias stress tests — position swaps, verbose-card inflation, and empty-reference analogs of the kind that cause the >40% collapse reported in the survey — plus a transitivity audit of pairwise rankings following Investigating Non-Transitivity in LLM-as-a-Judge. Experiment 4 (transfer) is the supervised-versus-judge cross-context test described above. A final extension applies the judge as a safety and coherence filter on de novo candidates from generative systems such as CURE and GxVAEs, whose outputs are currently vetted only by regression proxies and string validity — but this is contingent on the judge earning its place in Experiments 1–3.
The decisive check for the whole program is simple: if the structured judge does not improve phenotypic prediction beyond the best supervised baseline under context transfer, and does not improve potency ranking beyond sRGES, then semantic adjudication of signature reversal is a dead end, and the benchmark will at minimum have made that failure measurable rather than assumed.
References
- Reversal of cancer gene expression correlates with drug efficacy and reveals therapeutic targets
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
- A Survey on LLM-as-a-Judge
- From Knowledge to Treatment: Large Language Model Assisted Biomedical Concept Representation for Drug Repurposing
- A foundation model for clinician-centered drug repurposing
- Reading the Cell, Designing the Cure: Perturbation-Conditioned Molecular Diffusion for Function-Oriented Drug Design
- GxVAEs: Two Joint VAEs Generate Hit Molecules from Gene Expression Profiles
- Combinatorial prediction of therapeutic perturbations using causally inspired neural networks
- In silico biological discovery with large perturbation models
- Projecting genetic associations through gene expression patterns highlights disease etiology and drug mechanisms
- Common and cell-type specific responses to anti-cancer drugs revealed by high throughput transcript profiling
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Investigating Non-Transitivity in LLM-as-a-Judge