When two trained coders map the same clinical note to a medical vocabulary and pick different codes, the disagreement can mean two different things. The coders may agree on the kind of problem and differ only on how specific to be, as with two flavors of depression. Or they may file the same complaint in unrelated branches of the vocabulary, one under psychiatric disorders and the other under social circumstances, because the vocabulary has no natural place for it. Standard agreement statistics score both cases as a plain mismatch. This proposal builds an agreement measure that separates them and uses it to test a specific suspicion: that adverse-event vocabularies quietly assume hormonal contraceptives are "behaviorally inert," so the mood, libido and relationship effects patients report have no coherent home in the vocabulary.
The suspicion has a precedent in another field. Wildlife managers long treated the immunocontraceptive PZP as a switch that blocks conception and does nothing else. Field studies of feral horses then showed that treated mares changed social groups and visited more groups than controls (Nuñez et al. 2009). The disruption did not stop when treatment stopped (Madosky et al. 2010), and it grew with the number of doses (Nuñez et al. 2017). In people, an analysis of electronic health records found that mood changes and weight gain were among the most common reasons for switching contraceptives (Identifying Reasons for Contraceptive Switching). That study grouped reasons with unsupervised topic clustering, not a standard vocabulary, and checked them with a single reviewer. Nobody has asked whether the formal vocabularies used in drug-safety surveillance can represent these reasons consistently. Disagreement among coders, measured properly, is a direct way to find out.
Core Claim
The claim has two parts. First, a chance-corrected agreement coefficient that uses the vocabulary's own hierarchy to measure how far apart two codes are can be split into two pieces. One piece is granularity disagreement: the coders land in the same organ-system branch but at different depths or on sibling terms. The other is placement disagreement: they share no top-level branch. Second, for contraceptive behavioral and relational effects, placement disagreement makes up a much larger share of total disagreement than it does for somatic contraceptive effects such as bleeding or weight change, or for a drug class whose side effects are physical (statins). It should also survive a round of sharpened annotation guidelines. If both parts hold, the vocabulary itself is the problem, which is a measurable instance of the inertness assumption. If placement disagreement falls once guidelines improve, the problem was the instructions, and the vocabulary is exonerated.
The vocabulary in question is MedDRA (the Medical Dictionary for Regulatory Activities), the coding scheme used in drug-safety reporting. Two of its properties matter here. Its specific "Preferred Terms" roll up into broad "System Organ Classes," the organ-system branches at the top of the hierarchy. And a single Preferred Term can sit under more than one System Organ Class, so the hierarchy is a graph rather than a strict tree. This multi-parent structure and the fact that a single note often gives several reasons are what make a naive per-level analysis awkward. Both are handled explicitly below.
Research Program
The agreement measure. The measure starts from Krippendorff's alpha, a chance-corrected agreement statistic that accepts any distance between labels, $\alpha = 1 - D_o/D_e$. Here $D_o$ is the average observed distance between labels that different coders assigned to the same item, and $D_e$ is the average distance expected if labels were paired at random from the pooled label distribution. Skjærholt plugged tree edit distance into alpha to score agreement on syntax trees and showed that it separates structural errors from label errors where flat accuracy does not (A chance-corrected measure of inter-annotator agreement for syntax). We reuse that method in a setting it was never applied to, a medical vocabulary.
Our label distance is computed on the MedDRA hierarchy. For two terms $a$ and $b$, $\delta(a,b) = 1 - d(a,b)/L$, where $d(a,b)$ is the depth of their deepest shared ancestor, maximized over the multiple parent paths, and $L$ is the depth of the hierarchy. Items that receive several labels are compared with an optimal one-to-one matching between the two coders' label sets, with unmatched labels charged the maximum distance.
The split works pair by pair. Any coder-pair distance whose two labels share no System Organ Class on any parent path counts toward placement disagreement, $D_o^{place}$. Everything else counts toward granularity disagreement, so that $D_o = D_o^{gran} + D_o^{place}$. The quantity we report per reason category is the placement share $\rho = D_o^{place}/D_o$. Confidence intervals come from bootstrapping over items.
The corpus. We build a new multi-coder corpus, since no public resource on Lacuna has contraceptive behavioral effects coded to MedDRA by several people. Passages come from three places. The first is MEDLINE sentences in the PHEE pharmacovigilance corpus that mention hormonal contraceptives; PHEE annotates who took a drug, the treatment and the effect, but it is scored by token F1 and does not single out behavioral reactions (PHEE). The second is MIMIC-IV clinical notes describing contraceptive discontinuation. The third, if a data-use agreement allows, is the UCSF switching notes. A matched set of statin passages serves as the somatic-effect control; the Pravastatin study's myalgia and dyspepsia signals (Liu & Aickelin) show that this class yields clean physical reactions.
We target roughly 600 contraceptive and 300 statin passages. Four coders label each one: two MedDRA-certified coders, one gynecologist and one pharmacovigilance specialist. Each coder assigns MedDRA Preferred Terms, may flag "no adequate term," and records whether the effect is attributed to the drug, pre-existing, or unclear. The attribution field follows PHEE's finding that separating an existing disorder from a drug effect is a persistent failure point. It also echoes the horse finding that behavioral effects outlast treatment, which makes the timing of attribution genuinely hard.
Guideline revision. After the first round, we write sharpened guidelines targeted at the observed disagreements. A disjoint set of 200 passages is then annotated under the new rules. This round decides whether placement disagreement is a property of the vocabulary or of the instructions.
Language models as coders. We then treat GPT-4 zero-shot coding, following the UCSF prompting setup, as a fifth rater. We ask whether its disagreements with humans fall inside the human disagreement distribution, meaning similar placement shares, or outside it. LLM-codex reports that zero-shot GPT-4 over-assigns non-specific codes (Surpassing GPT-4 Medical Coding with a Two-Stage Approach), which predicts excess granularity disagreement toward shallow terms. The hierarchical measure can test that prediction where flat F1 cannot.
Baselines
The naive reference point is raw percent agreement at the Preferred Term level. It is what most clinical annotation reports give, and it shows whether chance correction changes any conclusion, given how skewed reason frequencies are.
The first real baseline is flat nominal Krippendorff's alpha, which uses a 0/1 distance on the same multi-coder matrix and the same set matching. It is the standard practice that the MDACE evidence corpus (MDACE) and the UCSF study both stop short of reporting.
The strongest competitor is per-level agreement: Fleiss' kappa computed separately at the Preferred Term level and the System Organ Class level. For a strict tree, it would separate granularity from placement almost as directly as our split does. We run it on identical data. Multi-parent terms are assigned to MedDRA's designated primary organ class, and multi-label items are handled with the same set matching. We count our measure as beating per-level kappa only if three things hold. On the synthetic corruption benchmark, it ranks corruption types correctly where per-level kappa does not. On the real corpus, it flags category differences in placement share that per-level kappa misses. And that gap comes specifically from multi-parent terms. If per-level kappa reaches the same conclusions about behavioral versus somatic effects, we will report that. The contribution then becomes the empirical finding plus the corpus rather than the coefficient.
Two ablations isolate what the hierarchy contributes. The first replaces the hierarchy distance with cosine distance between sentence embeddings of term names, which tests whether expert structure matters or whether any notion of semantic closeness would do. The second removes the multi-parent maximization and uses only primary paths, which tests whether the multiaxial structure drives the result.
For the language-model rater, the strongest reported number in this domain is GPT-4 zero-shot accuracy of 91.4% on switching reasons, with micro F1 of 0.849 for started medications and 0.881 for stopped medications (Identifying Reasons for Contraceptive Switching). Those figures concern flat reason categories. For contraceptive effects coded to MedDRA, no reported result was found on Lacuna. As a broader reference, full ICD-10 coding on MIMIC-IV reaches a best macro F1 of only 5.42% (Mimic-IV-ICD). That shows how much flat leaf-level scoring hides when predictions land near, but not on, the gold code.
Why not just…?
Why not just compute kappa at each level of the hierarchy? This is the real threat, and it is run as the main baseline rather than dismissed. Per-level kappa works cleanly when every term has one parent and every item one label. MedDRA breaks the first condition, and switching notes break the second. Forcing primary paths can make two coders who chose the same multi-parent term appear to agree at every level, while two coders who chose different terms sharing a secondary organ class appear to disagree on placement. Skjærholt's synthetic study found that level-blind scores degrade badly when structural and label noise co-occur. Our corruption benchmark checks whether that carries over to MedDRA. If it does not, we drop the claim of a better coefficient.
Why not just use an LLM with topic clustering, as the UCSF study did? Topic clusters cannot be pooled with drug-safety reports, which are coded in MedDRA. Clusters also have no fixed hierarchy, so the question this proposal asks, whether the vocabulary has a coherent place for behavioral effects, cannot even be posed. The language-model rater experiment additionally tests whether zero-shot coding inherits the shallow-code bias that LLM-codex documented.
Why not just write clearer guidelines and adjudicate to consensus? This may be the correct explanation, so it is built in as an experiment rather than argued against. The second annotation round uses sharpened guidelines. If placement share for behavioral reasons drops to the somatic level, the ontology-misfit hypothesis is refuted. Prior evidence leans the other way. Martínez Alonso et al. showed that annotator confusion tracks real category ambiguity: training a tagger to penalize errors less on classes humans often confuse raised one Danish category's F1 from 0.0 to 16.53 (Supersense tagging with inter-annotator disagreement). MDACE found that certified coders diverge systematically by role. Neither result settles the contraceptive case, though.
Proposed Experiments
The first experiment validates the measure. Following Skjærholt's corruption design, we take gold MedDRA codes from the corpus and gold ICD-10 codes from MIMIC-IV, then inject controlled noise of three kinds: sibling swaps, swaps within the same organ class, and swaps across organ classes, at several rates. A good measure must order these corruptions correctly and track the injected placement fraction. Flat alpha, per-level kappa and the embedding-distance variant are scored on the same corrupted sets.
The decisive test runs on the real corpus. We compare placement share for behavioral and relational contraceptive reasons against somatic contraceptive reasons and against statin effects, with bootstrap intervals. Three corroborating signals should line up with high placement share if the misfit is genuine: the rate of "no adequate term" flags, the spread of chosen terms across organ classes, and attribution disagreement. The hypothesis survives only if the behavioral gap persists in the revised-guideline round. A null result is equally reportable: it would mean the vocabulary already accommodates these effects and that earlier gaps lay in practice, not structure.
The third experiment places GPT-4 within the human disagreement distribution. We test whether its placement share and preference for shallow terms differ from the human coders' using a permutation test that treats it as an exchangeable rater.
A last downstream check reuses the Martínez Alonso recipe in a modest form. We fine-tune a clinical encoder to code the passages under three losses: uniform, hierarchy-distance-weighted, and weighted by the observed disagreement pattern. Macro F1 on behavioral reasons, scored under both flat matching and our hierarchical measure, shows whether the measured disagreement structure is useful for training as well as for diagnosis.
References
- Immunocontraception decreases group fidelity in a feral horse population during the non-breeding season
- The effects of immunocontraception on harem fidelity in a feral horse (Equus caballus) population
- Lingering effects of contraception management on feral mare (Equus caballus) fertility and social behavior
- Identifying Reasons for Contraceptive Switching from Real-World Data Using Large Language Models
- A chance-corrected measure of inter-annotator agreement for syntax
- PHEE: A Dataset for Pharmacovigilance Event Extraction from Text
- Detect adverse drug reactions for the drug Pravastatin
- Surpassing GPT-4 Medical Coding with a Two-Stage Approach
- MDACE: MIMIC Documents Annotated with Code Evidence
- Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification
- Supersense tagging with inter-annotator disagreement