Most large clinical text datasets are labeled by programs, not people. A rule-based labeler reads each radiology report and decides whether findings such as edema or effusion are present, absent, or hedged. A neural model is then trained on those noisy labels, and a few hundred expert annotations patch the remaining errors. Every current system of this kind assumes that each labeling source is equally reliable everywhere. That assumption is doubtful. A negation rule tuned on Stanford's report templates can misread Boston's, and a phrase like "cannot be excluded" can carry different weight for different radiologists, pathologies, and patient groups. We propose to let the user declare, alongside the extraction schema, the population groupings along which reliability might vary: hospital site, report template, age band, insurance type. A compiler then fits a label model in which each source's accuracy is partially pooled across those groups. Groups with plenty of data get their own accuracy estimates. Small groups borrow strength from the population, to the degree the data show they resemble it.
The practical stakes are the worst-off subgroups. Aggregate extraction F1 is already close to radiologist agreement, but aggregate numbers hide which patients' reports are mislabeled. Those mislabels flow into every image model and cohort study built on top. Partial pooling targets that error directly and makes it auditable: the compiler reports, per group and per source, how trustworthy the labels are, with uncertainty attached.
Core Claim
In weak supervision, the source of population heterogeneity that matters most is the labeling sources themselves, not the end model. How often a labeler is right depends on group. Treating that dependence as a random effect in the label model should reduce label error in small, stylistically distinct groups more than any end-model remedy, such as extra capacity per group or worst-group training objectives. Those remedies cannot repair labels that are systematically wrong for a group.
The label model is the part of a weak-supervision system that learns how accurate each noisy source is without ground truth. Concretely, let $\theta_{jgk}$ be the log-odds that source $j$ assigns the correct label for class $k$ in group $g$. We model
$$\theta_{jgk} = \mu_{jk} + u_{jgk}, \qquad u_{jgk} \sim \mathcal{N}(0, \tau_{j}^2),$$
where $\mu_{jk}$ is the population accuracy of source $j$ on class $k$, and $u_{jgk}$ is a group-specific deviation whose spread $\tau_j$ is learned per source. Setting $\tau_j \to 0$ recovers the global label model used by Overton. Setting $\tau_j \to \infty$ recovers an independent label model per group. The novelty lies in estimating where each source sits between those extremes, and in exposing the grouping keys as a declarative field so that engineers do not hand-code this per cohort.
Relation to Prior Work
Three lines of work motivate the gap. Overton compiles a declarative schema into a multitask model, learns source accuracies without labels, and lets engineers declare "slices" that receive extra model capacity. Its source accuracies are global, and its slices act on the end model, not on label quality. We keep its schema-plus-sources interface and move heterogeneity into the label model. Declarative Learning-Based Programming names robustness across heterogeneous environments as an unmet requirement for declarative AI systems but gives no mechanism; we supply one narrow, testable mechanism. The declarative extraction systems SystemT and PalimpChat optimize how extraction operators or LLM calls execute. They do not model how operator reliability varies across document populations.
On the clinical side, CheXpert found that the best way to treat hedged labels differs by pathology: mapping "uncertain" to positive worked for atelectasis and edema, while a separate uncertain class worked better for cardiomegaly. That is direct evidence that label semantics are heterogeneous, and we treat pathology as one axis of pooling. CheXbert trains BERT on about 200,000 rule-labeled reports and then fine-tunes on about 1,000 radiologist-labeled ones. It supplies our strongest baseline and our recipe for using expert labels, but it uses a single labeler with one global reliability. RadGraph documents that performance drops moving from MIMIC-CXR to CheXpert reports and that radiologists disagree on relations (Cohen's kappa 0.397 on CheXpert). This constrains us: some between-site "error" is genuine ambiguity, so our evaluation must separate the two. CheXpert Plus provides 187,711 reports with eight demographic axes and RadGraph annotations, which makes the grouping keys available at all.
On the heterogeneity side, To Pool or Not To Pool shows that pooled training under group-fair objectives lowers minority-group risk relative to isolated models. Its evidence comes from 15-dimensional synthetic logistic regression with groups of 75%, 20%, and 5%. We borrow its question, whether pooling helps the 5% group, and test it where the pooled object is source reliability in real clinical text. Hierarchical random-effects models such as Hierarchical Dirichlet Processes with Random Effects and shared-structure estimators such as Brain covariance selection supply the statistical template: shared components whose parameters shift by group. Finally, Population heterogeneity in clinical cohorts warns that real groupings are latent and intersecting. For that reason the compiler selects among candidate grouping keys rather than trusting a single declared partition.
Research Program
The system has three parts. The first is the declaration. The user specifies the output schema (the 14 CheXpert observations, each positive, negative, uncertain, or blank), the labeling sources, and candidate grouping keys. The keys are site, report section template (recoverable from the 11 sections in CheXpert Plus), age band, sex, race, and insurance type, plus pathology, which is always a key.
The second part is the sources. We use five of them, chosen to differ plausibly in how their reliability depends on group:
- the CheXpert rule labeler;
- a keyword-and-negation dictionary;
- observation labels derived from a RadGraph-style entity extractor, mapping "definitely present," "uncertain," and "definitely absent" onto the schema;
- a few-shot LLM extractor run with the self-verification pipeline of Self-Verification Improves Few-Shot Clinical Information Extraction, which quotes evidence and prunes unsupported findings;
- CheXbert itself, used as a source when it is not the model under test.
The third part is compilation. The label model above is fit by variational inference on unlabeled reports. It uses the same conditional-independence assumptions as standard label models, and the small expert set enters as observed labels in a semi-supervised likelihood. The compiler chooses which grouping keys receive random effects by held-out likelihood on the expert development split. It then trains a single BERT-class end model on the resulting probabilistic labels. It also emits an audit table of posterior source accuracy per group with credible intervals.
The analytical deliverable is an empirical answer to three questions. First, how large is between-group variation in source accuracy, for which sources, and along which keys? Second, does that variation, rather than end-model capacity, explain worst-group extraction error?
Baselines
The naive reference point is the CheXpert rule labeler used directly as the extractor. CheXbert reports it at 0.743 average F1 against radiologist annotations. The same labeler reports 0.969 micro-F1 in CheXpert under a different reference standard, so we rerun it on our own splits and report only those numbers.
The strongest existing method is CheXbert, which reports 0.798 average F1 against a radiologist benchmark of 0.805 in Combining Automatic Labelers and Expert Annotations. We retrain it with its two-stage recipe on our data. Stage two uses exactly the same expert-labeled reports our method sees, with backtranslation as in the original and the same hyperparameter search budget. Beating it means higher worst-group F1 across declared groups with no loss in average F1 beyond a pre-set margin of 0.005. A gain in average F1 alone does not count.
The self-verified LLM extractor is run as a standalone system with the same few-shot budget. We found no reported result on Lacuna for self-verification on CheXpert or MIMIC-CXR report labeling. Its closest reported numbers are on other tasks: MIMIC-IV ICD-10 F1 rising from 0.487 to 0.533, and trial-arm F1 from 0.419 to 0.530, both with GPT-4.
Two ablations isolate the key ingredient. The first is a global label model with $\tau_j = 0$. This is Overton-style weak supervision over the same five sources, with Overton-style slices on the end model for every declared group. The second is a fully isolated label model per group. If partial pooling does not beat both on worst-group F1, the central claim fails.
A further baseline applies worst-group (group-robust) training to the end model using global labels. Tuning budgets are matched across all systems, and all end models share the same architecture and compute.
Why not just…?
Why not just fine-tune CheXbert on a few expert labels from each site? This is the most natural fix. It corrects errors only where expert labels exist, and those are scarcest in the smallest groups, the ones that matter. Our design gives both methods the identical expert budget, stratified by group. If per-site fine-tuning matches us on the smallest group, that is a real refutation, and we will report it as such. We expect it to fall behind where a group is small but its reports number in the thousands. There, partial pooling exploits unlabeled volume and fine-tuning cannot.
Why not just use Overton-style slicing or worst-group training on the end model? Both reallocate model effort toward a group but train on the same labels. If a source systematically misreads one site's hedging, extra capacity learns the error faithfully. The ablation with global labels and per-group slices tests this directly. The diagnostic is whether worst-group error correlates with estimated per-group label error, which end-model methods cannot change.
Why not just run a strong LLM with self-verification on every report and skip weak supervision? This is a real threat. If a self-verified LLM is accurate and uniformly so across groups, the label model adds little beyond cost savings. The same architecture measures this: the LLM is one source, and its estimated $\tau_j$ tells us whether its reliability varies by group. We also report it standalone on the stratified expert set, along with inference cost for the 187,711-report corpus. The verification passes inflate token counts, as Self-Verification Improves Few-Shot Clinical Information Extraction acknowledges. If its worst-group F1 matches ours at acceptable cost, we will say the contribution reduces to an auditing tool.
Proposed Experiments
The data are CheXpert Plus reports, which carry demographics, and MIMIC-CXR reports, which serve as a second site. Expert labels come from three places: the radiologist annotations used by CheXbert, the RadGraph development set (500 MIMIC-CXR reports) and double-annotated test set (100 reports), and a new annotation of roughly 600 CheXpert Plus reports stratified by site, template, and demographic cell, with small cells oversampled. That set is split into a development half, used for compilation and for stage-two fine-tuning, and a test half used only for scoring. Double annotation on a subset estimates radiologist agreement per group. We report per-group F1 against the ceiling of per-group inter-annotator agreement, not raw F1, so that groups with intrinsically ambiguous reports are not scored as failures.
The first decisive check is calibration of the audit. Posterior per-group source accuracies, estimated largely without labels, must match accuracies measured on the held-out expert set, as judged by rank correlation and interval coverage. Otherwise the pooling is fitting noise.
The second check is the pooling curve. We subsample one site or demographic cell down to 5% of the corpus, mirroring the smallest group in To Pool or Not To Pool, and plot that group's label error and end-model F1 under global, isolated, and partially pooled label models. The prediction is that partial pooling dominates both extremes at small group sizes and converges to isolation as the group grows.
The third check is attribution. Across groups, the gain from partial pooling should track the estimated between-group spread $\tau_j$ of the sources that group relies on. If gains appear where $\tau_j$ is near zero, the mechanism is not what we claim.
We commit in advance to one negative outcome. If all learned $\tau_j$ are near zero, reliability does not vary meaningfully by group. The method then collapses to Overton, and we will report that clinical label heterogeneity lives elsewhere.
References
- Overton: A Data System for Monitoring and Improving Machine-Learned Products
- Declarative Learning-Based Programming as an Interface to AI Systems
- SystemT: An Algebraic Approach to Declarative Information Extraction
- PalimpChat: Declarative and Interactive AI analytics
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
- Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats
- To Pool or Not To Pool: Analyzing the Regularizing Effects of Group-Fair Training on Shared Models
- Hierarchical Dirichlet Processes with Random Effects
- Brain covariance selection: better individual functional connectivity models using population prior
- Population heterogeneity in clinical cohorts affects the predictive accuracy of brain imaging
- Self-verification improves few-shot clinical information extraction