Learning Which Clinical Extractor to Trust, Per Note Type, at a New Hospital

Hospitals that want structured facts out of free-text notes now have many extractors to choose from. These include span-tagging encoders, graph parsers trained on radiology reports, and large language models prompted to read a note and list what they find. No single extractor wins everywhere. One extractor may be the most accurate at tagging medications in discharge summaries and still be a poor choice for relations in chest X-ray reports. Which one is best can also change when the same system moves to another hospital's notes. This proposal builds an ensemble that keeps a separate trust score for every combination of extractor, note type and entity type. When the ensemble arrives at a new hospital, it updates those scores from a small, deliberately chosen set of clinician-verified documents.

The work borrows from three areas. Benchmark design supplies the rule that results must be broken down by kind of application, because an aggregate score hides the cases that matter. Adaptive ensemble weighting supplies the method: model weights that move when a model starts making errors. Clinical text extraction supplies the setting where both ideas are needed. The central claim can be tested and could turn out false. It says that when an extractor moves between hospitals, the ranking of extractors changes in some note-type and entity-type combinations, and that sparse-label adaptation inside those combinations recovers accuracy more cheaply than fine-tuning a single model with the same labels. If the rankings do not change, static per-category routing is enough and the adaptive part is unnecessary. The first experiment is designed to find out.

What prior work establishes and leaves open

Evidence that no single extractor dominates comes from several benchmarks.

  • Clinical NER benchmark. The Named Clinical Entity Recognition Benchmark maps four corpora onto a shared clinical vocabulary, the OMOP data model. On span F1 the GLiNER encoder (65.67) beats GPT-4o (53.35) and Llama-3-70B (49.40). It evaluates each model separately and never combines them.
  • EHRNoteQA. EHRNoteQA finds the reverse pattern on long, multi-admission discharge summaries, where GPT-4 reaches 97.16% multiple-choice accuracy. Taken together, the two benchmarks suggest the better architecture depends on the task and the document type.
  • MedSecId. MedSecId shows large accuracy differences across five note types and 50 section classes: micro F1 0.955 but macro F1 0.787. This is the kind of stratification the benchmarking literature calls for.
  • Benchmarking literature. The work of Jia et al. on Hadoop workloads and BLOCKBENCH argues that one representative workload cannot predict behaviour on other workloads. Cross-corpus coreference work shows that the features a model relies on change completely from one corpus to another.
  • RadGraph. RadGraph supplies the cross-site case. A model trained on MIMIC-CXR (Beth Israel) reaches 0.94 NER F1 and 0.82 relation F1. On the external Stanford set, CheXpert, even radiologists agree only weakly on relations (κ = 0.397).

All of these papers motivate the proposal: they show that extractor quality is uneven across the workload. None of them combines extractors.

The method comes from adaptive ensembles.

  • MSAW. MSAW works on electronic health record data for RSV detection. It multiplicatively penalizes historical models that misclassify new encounters and gives the local model more weight as data accumulates. This raises AUROC to 0.870, compared with 0.728 for equal weighting.
  • Fed-POE. Fed-POE uses multiplicative weights over model snapshots and comes with regret guarantees.
  • CSGE. CSGE blends a model's global performance with its performance on nearby examples. It finds those examples by nearest-neighbour search in a reduced feature space (principal component analysis).

Each of these papers also limits the design. MSAW treats text extraction as a fixed preprocessing step (MetaMap features feeding a binary classifier). It assumes a fast stream of labels. MSAW and Fed-POE weight models by scalar prediction loss, not by structured span outputs. CSGE's nearest-neighbour step was tested only on small tables, and its authors warn that the neighbourhood becomes unreliable in high dimensions. That warning is the reason this proposal defines "local context" by explicit note-type and entity-type categories instead of embedding distance.

Two clinical pipelines are the closest existing designs. Self-Verification runs a single large language model through a fixed chain: extract, look for omissions, ground each item in source text, then prune. Entity Decomposition with Filtering splits broad entity types into narrower ones, queries open NER models, and applies a fixed language-model filter. Both are fixed chains. Neither learns from deployment feedback which source deserves trust, or for which category.

Core Claim and Mechanism

Each base extractor $m$ proposes typed spans, and where applicable relations, for a note. Candidates from different extractors are aligned by overlap. A candidate is accepted when the weighted vote of the extractors proposing it passes a threshold. The weights depend on the cell $c$, defined as the pair (note type, entity or relation type). Note type comes from metadata, or from a MedSecId-trained section tagger when headers are missing.

After the clinician verifies a batch of documents, each extractor's weight in a cell is updated multiplicatively from its span-level loss $\ell_{m,c}$ on that batch. The loss is the share of the extractor's spans in cell $c$ that were false positives or false negatives. The weight used for voting is a shrunk mix of the cell-level weight and the extractor's global weight:

$$w_{m,c} \leftarrow w_{m,c}\, e^{-\beta \ell_{m,c}}, \qquad \tilde w_{m,c} = \lambda_c w_{m,c} + (1-\lambda_c) w_m, \quad \lambda_c = \frac{n_c}{n_c+\tau}.$$

Here $\beta$ is the learning rate, $w_m$ is the extractor's weight pooled over all cells, $n_c$ is the number of verified spans seen in cell $c$, and $\tau$ sets how quickly the weights trust cell-specific evidence. The update is the exponential-weights rule behind Fed-POE's regret analysis. The shrinkage plays the role of CSGE's balance between global and local performance. It keeps rare cells, which the MedSecId long tail shows are common, from collapsing onto whichever extractor got lucky on a few examples.

Because labels are scarce, the system does not verify random documents. Each round it selects the notes where the extractors disagree most, weighted by how uncertain the current cell weights are. This responds directly to the feedback-delay limitation that MSAW leaves open.

Research Program

The base pool covers the architecture families the file shows to be complementary:

  • a BioBERT token classifier;
  • GLiNER-multitask-large;
  • DYGIE++ with PubMedBERT trained on RadGraph's MIMIC split;
  • the open NER models UniversalNER and GNER;
  • Llama-3-70B-Instruct, prompted with the Self-Verification pipeline.

All extractors are frozen. The only things learned are the cell weights and one acceptance threshold per cell.

The evaluation follows the benchmarking principle that every result must be broken down. Every number is reported per cell and per site, alongside the aggregate. The study also reports a rank-flip statistic: the fraction of cells where the best extractor on source-site validation data is not the best on the target site.

Baselines

Do nothing. The naive reference is the single extractor that scores best on source-site validation data, deployed unchanged at the target site. This is what most groups would do. For radiology entities and relations the natural candidate is the RadGraph DYGIE++ PubMedBERT model, which reports 0.94 NER micro F1 and 0.82 relation F1 on MIMIC-CXR (RadGraph). No reported result was found on Lacuna for that model's extraction F1 on CheXpert, so we measure it ourselves. For general clinical NER the reference is GLiNER at 65.67 span F1 (Named Clinical Entity Recognition Benchmark). We reproduce these numbers before comparing anything.

Uniform voting. All extractors vote with equal weight. This mirrors the equal-weighting baseline that MSAW beat by 0.14 AUROC.

Static per-cell routing. For each cell, use the extractor that was best on source validation data. This is the strongest competitor that requires no learning at the target site, because it already uses the cell structure.

Fine-tuning on target labels. The strongest competitor that does use target labels is to fine-tune the best single encoder on exactly the verified target documents the ensemble receives. It gets the same label budget, the same number of hyperparameter trials, and the same GPU hours as tuning the ensemble's $\beta$ and $\tau$.

Language-model aggregation. Following Self-Verification, the strongest prompting-based competitor merges all extractors' candidates with Llama-3-70B and then prunes them through evidence grounding.

Ablations. Two ablations isolate the key ingredient. The first removes the cells and keeps only global multiplicative weights, which is effectively MSAW or Fed-POE applied to spans. The second keeps the cells but freezes their weights at source values. A third ablation replaces disagreement-based document selection with random selection.

What counts as success. The method wins if it beats the best of fine-tuning and static routing on target-site macro F1 across cells, at matched label budgets of 50, 200 and 500 verified documents. The gap should exceed the spread across random seeds. It should also not reduce F1 in more than a small minority of cells.

Why not just…?

Why not just route each note type to whichever extractor won on validation? This is the most serious threat. If extractor rankings stay the same across sites, static routing captures almost all the benefit, and the adaptive component adds nothing. The rank-flip statistic decides the question directly. The RadGraph result that even radiologists annotate CheXpert relations differently (κ = 0.397) suggests rankings may shift at least for relation cells, but that is a hypothesis, not evidence. If fewer than roughly a tenth of the cells flip and static routing matches the adaptive ensemble, we will report that the useful contribution is the per-cell evaluation and routing, not the online weighting.

Why not just fine-tune the best extractor on the target-site labels? With a few hundred verified notes this could work well, especially for common cells. Our matched-budget comparison will show it. We expect the ensemble to have an advantage in rare cells and at the smallest budgets, where fine-tuning has too few examples per category, and fine-tuning may win at the largest budget. A crossover point is a useful result in itself, because it tells a hospital when to switch strategies.

Why not just let a large language model judge and merge everyone's outputs? Self-Verification and Entity Decomposition with Filtering both show that a language-model filter improves precision. However, the clinical NER benchmark shows decoder models trailing encoders on span F1 by about 12 points. A judge that is itself weaker at span boundaries may systematically override the better extractor. The language-model aggregation baseline tests this. Combining the two, with learned cell weights as a prior for the judge, is a natural extension if the judge proves complementary.

Proposed Experiments

Experiment 1: cross-site radiology. Train or calibrate the system on RadGraph's MIMIC-CXR split, then stream CheXpert reports. Within each round, the system receives verified labels for the selected documents before it processes the next batch. Accuracy is scored on held-out CheXpert annotations.

Experiment 2: shift across corpora. Stream the four OMOP-mapped corpora (NCBI Disease, CHIA, BC5CDR, BIORED) in sequence. Each switch between corpora acts as a deployment shift, and the corpus plays the role of the note-type axis of a cell.

Experiment 3: note-type diversity. Using MIMIC notes with MedSecId sections as context, stratify the CheXbert observation-labeling task on MIMIC-CXR/CheXpert. This tests whether section-derived cells help. CheXbert's reported 0.798 average F1, against 0.805 for radiologists, is the single-model reference.

Metrics. We report per-cell and macro span F1 and relation F1, worst-cell F1, and labels needed to reach the source-site F1. Because language-model extractors are expensive, we also report compute per note.

Decisive checks. Three results matter most. First, the rank-flip rate between sites. Second, the matched-budget gap between the adaptive ensemble and target fine-tuning. Third, the gap between the cell-conditioned ablation and the global-weights ablation, which isolates whether the gain comes from per-cell trust or simply from having several models.