Medication recommendation models read a patient's diagnosis and procedure codes and propose a set of drugs. They are scored by how much that set overlaps with the one prescription written in the chart. Medicine rarely has a single right answer, though. A patient whose recorded regimen includes one ACE inhibitor could equally well have received another, and a model that proposes the alternative is counted as wrong twice: once for the missed drug and once for the "false" drug. This note proposes an evaluation, and a matching training objective, in which each test visit has a set of acceptable prescriptions rather than one. The set is built by substituting recorded drugs with alternatives that a biomedical knowledge graph says serve the same indication, that similar patients in the data actually received, and that introduce no known drug-drug interaction.
The stakes are practical. Every recent system on the MIMIC-III recommendation benchmark reports Jaccard, F1 and precision-recall area against the chart, and reported gains are often a few percentage points. These include GAMENet, SafeDrug, COGNet, CIDGMed and HIER. If a meaningful share of each model's "errors" are clinically equivalent substitutions, the leaderboard may be measuring agreement with one physician's habits rather than clinical quality. Training against the single chart may also push models toward the common drugs that nearly everyone receives. This mirrors the bland "safe response" that one-to-many dialogue generation falls into.
Core Claim
Clinical recommendation is a one-to-many problem: one input permits several valid outputs. Its evaluation and training should use multiple references built from external knowledge plus empirical support from similar patients. Using only one of these sources fails in a predictable way. Knowledge-graph equivalence alone licenses substitutions that no clinician would make for this patient. Neighbor-patient evidence alone is too thin, because EHR visit vectors are extremely sparse (a visit activates perhaps 5–25 of tens of thousands of codes, per Recent Advances in Predictive Modeling with Electronic Health Records). In that regime near neighbors are scarce, and rare drugs get no alternatives at all.
The claim has three parts, and each can be tested. First, a multi-reference score built this way agrees better with pharmacist judgments than chart Jaccard does. Second, it changes the ranking or the size of the gaps among existing models. Third, training with a "best-matching reference" loss reduces the collapse toward generic high-frequency drugs without raising the interaction rate.
Relation to Prior Work
The motivating evidence comes from dialogue. There, Achieving Reliable Human Assessment of Open-Domain Dialogue Systems and Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings. show that single-reference overlap metrics can track human judgment poorly or even negatively. $Q^2$ evaluates a response by generating questions from it and answering them against a grounding document. It reaches a system-level Spearman correlation of 0.9798 with human consistency judgments, versus 0.3051 for BLEU. We borrow the principle of verifying outputs against external knowledge instead of a single reference, not the machinery. $Q^2$ works on free text, while our outputs are sets of structured drug codes, and the knowledge source is a drug graph rather than a Wikipedia passage.
The failure mode we target during training is described by Elastic Responding Machine and SC-Seq2Seq. Both show that cross-entropy against one target averages over valid continuations and yields generic outputs. They fix it with latent mechanisms or a manual specificity dial. We instead supply the model with the alternative targets explicitly, so there is nothing to average over.
The clinical systems constrain and supply the method. GAMENet and SafeDrug show how to penalize interactions from TWOSIDES. We use their interaction rate as a hard filter on reference sets and as a guardrail metric. COGNet is the closest existing work. It already generates prescriptions autoregressively with a copy-or-predict decoder, which acknowledges that prescribing is a sequence of choices, but it is trained and scored against the chart alone. CIDGMed shows that co-occurrence graphs carry spurious comorbidity links. For that reason our knowledge-graph equivalence relation is built from curated indication and target information, not from co-prescription statistics. G-BERT and GT-BEHRT supply the pretrained visit encoders we use to define patient similarity, since raw sparse vectors make nearly every patient an isolated outlier.
Research Program
The first component is the reference-set builder. For each recorded drug in a test visit, candidate substitutes come from DrugBank as drugs sharing an indication and a primary pharmacological target. We intersect this with the ATC hierarchy so that substitutes stay in a plausible therapeutic family. A candidate survives two further checks. It needs empirical support: among the $k$ nearest training patients in a pretrained visit-embedding space, restricted to those sharing the visit's principal diagnoses, at least $m$ received the candidate in place of the original. It also needs safety: substituting it introduces no TWOSIDES interaction with the rest of the regimen, using the full interaction set, not only the forty most severe types GAMENet modeled.
Where neighbors are too few, which is common for rare drugs, the builder falls back to knowledge-graph-only substitution. It marks those references as lower confidence, so we can report how many credits depend on the graph alone. The acceptable set $\mathcal{R}(v)$ for visit $v$ contains the chart regimen plus all single and pairwise substitutions that pass these checks.
The second component is the score. The multi-reference Jaccard of a prediction $\hat{y}$ is $\max_{r \in \mathcal{R}(v)} J(\hat{y}, r)$, where $J$ is ordinary set Jaccard. It reduces to the standard metric when $\mathcal{R}(v)$ contains only the chart. We report it alongside chart Jaccard, the interaction rate, and a genericity measure. The genericity measure is the fraction of predicted drugs that fall among the 20 most frequently prescribed drugs in training.
The third component is training. We replace the single-target multi-label loss with a best-match loss:
$$ \mathcal{L}(v) = \min_{r \in \mathcal{R}_{\text{train}}(v)} \ \text{BCE}(p_\theta(v), r) + \lambda \, \text{DDI}(p_\theta(v)). $$
Here $p_\theta(v)$ are the model's drug probabilities, $\mathcal{R}_{\text{train}}(v)$ is the reference set built from training data only, BCE is binary cross-entropy, and the interaction penalty is SafeDrug's controlled term with weight $\lambda$. Taking the minimum means the model must commit to one valid regimen rather than spread probability across all of them. This is the mechanism that should reduce genericity. The loss plugs into SafeDrug and COGNet backbones unchanged.
The fourth component is validation against people. Two clinical pharmacists independently rate a sample of about 300 visits, a scale comparable to the 300 human-rated pairs used by Ghazarian et al. For each visit they rate the regimens proposed by three systems as acceptable, acceptable-with-changes, or unacceptable, without seeing which system produced which. This is the ground truth against which both metrics are judged.
Baselines
The naive reference point has two parts. One is a frequency baseline that prescribes the most common drugs for the visit's principal diagnoses. The other is the untouched chart-Jaccard evaluation itself. If multi-reference scoring inflates the frequency baseline as much as the learned models, it is rewarding genericity rather than equivalence, and the metric has failed.
The strongest existing method is COGNet. Its authors report Jaccard 0.5336 ± 0.0011 and F1 0.6869 ± 0.0010 on the MIMIC-III multi-visit cohort (Conditional Generation Net for Medication Recommendation). It is the strongest competitor both on the leaderboard and conceptually, because its generative decoder is the architecture most likely to already produce valid alternatives. SafeDrug is the safety-focused comparator. Its paper reports a 19.43% interaction-rate reduction relative to LEAP and a 2.88% Jaccard gain over GAMENet. GAMENet, with reported Jaccard 0.4509 and F1 0.6081, anchors the older end.
We retrain all three from public code on the identical cohort split, with the same preprocessing and drug vocabulary. Each gets an equal hyperparameter search budget and five seeds. We first reproduce the reported chart-Jaccard numbers within noise, and only then compute the new scores. Beating COGNet under the proposal means that a COGNet backbone trained with the best-match loss improves pharmacist-rated acceptability over standard COGNet, at no higher interaction rate. Gains in multi-reference Jaccard alone do not count, since that metric is ours.
Two ablations isolate the key ingredient. The first builds reference sets from the knowledge graph only. The second builds them from neighbor-patient substitutions only. The claim that both are needed predicts that each ablation agrees with pharmacists worse than the combined builder. It also predicts that graph-only sets inflate scores for implausible substitutions, and that neighbor-only sets leave rare-drug visits with no alternatives.
Why not just…?
Why not just score at a coarser drug-class level, where substitutes collapse into one code? This is the most serious alternative. Collapsing to a class already credits many same-family swaps, and coarse aggregation is a well-worn response to sparsity. The problem is that classes are simultaneously too coarse and too fine. They credit swaps within a class that are wrong for a specific patient (for example, renal dosing constraints), and they miss equivalent drugs in sibling classes. PacRNN cautions that coarsening discards distinctions needed for personalized prediction. This is an argument, not proof. We therefore run class-level Jaccard as a third metric in the pharmacist study. If it agrees with pharmacists as well as our score does, the simpler metric wins and we will say so.
Why not just use nearest-neighbor patients' prescriptions as extra references, with no knowledge graph? This is data-driven and cheap. It is exactly the neighbor-only ablation. We expect it to fail on rare drugs and rare diagnosis combinations, where the sparsity described by MD-Manifold leaves few meaningful neighbors. It also inherits co-occurrence confounding, which CIDGMed documents. The ablation reports agreement with pharmacists separately for frequency strata, so a stratum-level win by the simple method would be visible.
Why not just have a large language model or clinicians judge every output? Clinician judgment is the gold standard we validate against, but it cannot be run on every test visit for every model and seed. An LLM judge is unverified for combinatorial drug sets and cannot be audited against specific interaction records. Our score is deterministic and traceable to graph edges and patient counts. The pharmacist study tells us whether that traceability costs accuracy.
Why not just add a diversity or specificity control, as in SC-Seq2Seq? Diversity is not the goal. A diverse wrong regimen is still wrong. Such controls also require a manually set dial, which SC-Seq2Seq's authors leave open. We include a COGNet variant that samples several regimens and keeps the most specific one. This tests whether generation-side tricks alone reduce genericity as much as the best-match loss does.
Proposed Experiments
All experiments use the MIMIC-III multi-visit cohort (6,350 patients) from MIMIC-III. Visit encoders are pretrained on the 30,745 single-visit patients, as in G-BERT. MIMIC-IV serves as a replication set. Interaction data come from TWOSIDES and drug structure and indication data from DrugBank.
The first decisive check concerns agreement. Across the roughly 300 pharmacist-rated visits, we compute the rank correlation between each metric and pharmacist acceptability. The metrics are chart Jaccard, class-level Jaccard, multi-reference Jaccard and the two ablations. There is no reported result found on Lacuna for pharmacist agreement with chart Jaccard on this benchmark. Establishing that number is itself a contribution.
The second check concerns ranking. We test whether GAMENet, SafeDrug and COGNet keep their order and gaps under the new score. We also measure the fraction of each model's chart-Jaccard "errors" that land in the acceptable set. If that fraction is below a few percent, the one-to-many concern is empirically minor for this benchmark, and we will report that negative finding plainly.
The third check concerns training. The best-match loss on SafeDrug and COGNet should lower the genericity fraction and raise pharmacist acceptability, with the interaction rate held at or below the baseline. Chart Jaccard is allowed to drop slightly, because the model is no longer rewarded for guessing the chart physician's specific choice. A robustness analysis perturbs the neighbor thresholds $k$ and $m$ and shows how much credit depends on graph-only references.
References
- GAMENet: Graph Augmented MEmory Networks for Recommending Medication Combination
- SafeDrug: Dual Molecular Graph Encoders for Recommending Effective and Safe Drug Combinations
- Conditional Generation Net for Medication Recommendation
- Dual-Granularity Medication Recommendation Based on Causal Inference
- Self-supervised Hierarchical Representation for Medication Recommendation
- Recent Advances in Predictive Modeling with Electronic Health Records
- Achieving Reliable Human Assessment of Open-Domain Dialogue Systems
- Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings.
- $Q^2$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
- Elastic Responding Machine for Dialog Generation with Dynamically Mechanism Selecting
- Learning to Control the Specificity in Neural Response Generation
- Pre-training of Graph Augmented Transformers for Medication Recommendation
- Graph Transformers on EHRs: Better Representation Improves Downstream Performance
- Pairwise-Ranking based Collaborative Recurrent Neural Networks for Clinical Event Prediction
- MD-Manifold: A Medical Distance Based Manifold Learning Approach for Heart Failure Readmission Prediction
- MIMIC-III, a freely accessible critical care database