Federated Report Structuring from Residents' Byproduct Feedback

Resident physicians do research in the time left over after patient care and paperwork, and there is very little of it. A German survey of rheumatology trainees found that documentation takes 31.8% of working hours and that 71.6% of research-active residents do their science in unpaid off-duty time (BEWUSST). One way to relieve this is to make the paperwork itself produce research-grade data. Language models can turn free-text clinical reports into structured records of which finding sits in which anatomical location, and those records are the raw material of retrospective studies. Patient data cannot leave the hospital, so such a model has to be trained with federated learning: each hospital trains the shared model on its own data and sends only model updates to a coordinator, which averages them. The obstacle is supervision. The strongest federated system for this task needs every site to hold expert-verified labels, and a site short on residents' time will not have them.

This note proposes to replace those expert labels with feedback that residents already give during normal work. When a resident reviews a machine-drafted structured report, they accept a field, reject it, or fix it in a few seconds. We ask whether these cheap signals, collected at different hospitals from annotators of uneven skill, are enough to train a federated extraction model that comes close to one trained on gold labels. Role conflict sets the constraint: the supervision budget is "whatever a resident gives as a byproduct of charting." Federated learning is the mechanism. The machine-learning question is new: how to do federated preference learning when the preference labels are sparse and noisy, and when the noise level differs from site to site.

Core Claim

The closest prior system is FedPref. It fine-tunes an 8-billion-parameter model with small low-rank adapters (LoRA) across six simulated hospitals. At each site it ranks candidate extractions from four frozen "teacher" models against the site's private ground truth, which turns them into chosen/rejected pairs, and then trains on those pairs with Direct Preference Optimization (DPO), a method that learns directly from such pairs. It reaches a gold-test F1 of 68.68 against a centralized upper bound of 71.67, and it lifts the worst site by 9.10 F1 over isolated local training. FedPref assumes each site already has correct structured labels.

We claim that most of that result can be kept when the ranking signal comes from per-field accept, reject, or edit actions on a few hundred reports per site, provided two things hold. First, each edit is converted into a preference pair: the corrected field is chosen and the original draft field is rejected. Second, each site's pairs are weighted by a locally estimated annotator reliability before the site trains its update. Reliability is estimated without any gold labels, by measuring how often a resident's accepts and rejects agree with the teacher ensemble's consensus on fields where the teachers are nearly unanimous.

The prediction to test is specific. Under noise that differs across sites, the reliability weighting should shrink the gap between the byproduct-trained model and gold-label FedPref much more than it helps under uniform noise. If it does not, the key ingredient is unnecessary and the simpler recipe wins.

Research Program

We first build a simulation of byproduct feedback on public radiology data. We use MIMIC-CXR reports, with labels from RadGraph (entities and "located at / suggestive of / modify" relations annotated by radiologists) and from Chest ImaGenome (anatomy-attribute scene graphs with a 500-patient gold subset). We split these into six sites following FedPref's design, with site sizes from 384 to 1,344 reports and deliberate skew in which diseases each site sees.

Simulated residents review model drafts field by field. Their responses are generated from the gold labels through a noise model with three parameters: the probability of missing an error (accepting a wrong field), the probability of a false rejection, and the probability of making an edit rather than just rejecting.

The noise levels are set to be realistic rather than flattering. RadGraph reports a Cohen's kappa of only 0.397 between expert radiologists on CheXpert report relations, so even experts disagree substantially. BEWUSST reports independent competence of 86.3% in one ultrasound skill and 43.1% in another, so skill clearly varies. Accordingly we give sites different noise levels, and within each site the noise differs by finding type.

We then train three variants. The first is FedPref with gold labels. The second uses byproduct feedback with uniform weighting. The third uses byproduct feedback with reliability weighting. All three share the same teacher ensemble, the same base model and LoRA configuration, and the same federated averaging.

The per-site feedback budget is varied from 50 to 800 reviewed reports. This traces how many resident-minutes of review buy how much F1.

As a secondary arm, we add differentially private noise to the adapter updates. Differential privacy is a formal guarantee that limits what the updates reveal about any single patient. Ming Li et al. found that 62% of healthcare federated studies omit such protection. The arm asks whether sparse, noisy supervision and privacy noise together push the model below usefulness.

Finally, we run a small reader study. Clinicians review structured drafts of de-identified MIMIC-CXR reports in two conditions, drafting from scratch and correcting a draft. We record seconds per report and the error rate after review. The purpose is to verify that "byproduct" feedback really costs seconds, not minutes. This does not measure burnout, and we do not claim it does.

Everything above is still a simulation on a single center's data, which is the "simulation trap" Ming Li et al. criticize: 91% of the healthcare federated studies they reviewed were simulations. We accept this for a first study and design the protocol so that it transfers unchanged to a real multi-hospital deployment.

Baselines

The naive reference point is isolated local training at each site on its own byproduct feedback, with no federation. FedPref's 9.10-point worst-site gap suggests local training will fail hardest at small sites. We run it with the same budget per site and the same number of DPO steps.

The no-training reference point is the base model run locally with schema-constrained decoding and a few examples in the prompt. It needs no feedback at all, so it measures whether any learning is worth the residents' clicks.

The strongest existing method is gold-label FedPref, which reports a gold-test F1 of 68.68 on its multi-hospital benchmark (centralized bound 71.67). We reimplement it on our split with identical data, model, adapter rank, rounds, and hyperparameter search budget. We do not compare to its published number directly, because our splits differ.

For relation-level evaluation, the reference extractor is DYGIE++ with PubMedBERT, which RadGraph reports at 0.94 entity micro-F1 and 0.82 relation micro-F1 on its MIMIC-CXR test set. It is trained centrally on 500 expert-annotated reports, so it shows what expert labels buy outside the federated setting.

The ablation that isolates our ingredient is uniform-weighted byproduct DPO compared with reliability-weighted byproduct DPO. A second ablation uses accept/reject signals only, with no edit-derived pairs.

Beating the strongest baseline does not mean exceeding gold-label FedPref. Success means recovering at least 80% of its improvement over isolated local training, on mean and worst-site F1, at a budget of 200 reviewed reports per site. The reader study must also show that reviewing a draft is faster than drafting from scratch.

Why not just…?

Why not just run a local open model zero-shot, with no training? This is the most serious threat, and it is cheapest. FedPref observed that its target model, learning only from its own outputs, produced identical candidates 71% of the time. That suggests untuned models handle fine-grained schemas poorly, but it is not a direct measurement of zero-shot accuracy. The no-training baseline settles the question. If it lands within two F1 points of the byproduct-trained model on worst-site relation F1, we drop the training component and redirect the work toward measuring review time on zero-shot drafts.

Why not just use rule-based labelers? The CheXpert labeler reaches 0.973 F1 on pleural effusion (Johnson et al.). Its output, however, is 14 report-level labels. Retrospective cohorts need where a finding is and whether it changed since the last exam, which is what RadGraph and ImaGenome relations encode. ImaGenome's own comparison-relation extraction reaches only 0.832 F1, so relations are the harder part. We report both label granularities so readers can see where rules suffice.

Why not just pool de-identified data centrally? Where pooling is legal, it is the better option. The centralized arm is our upper bound for that reason. The proposal targets the case Daniel Rueckert et al. and Ming Li et al. describe, where cross-institution transfer of records is blocked.

Why not just use standard noisy-label methods, such as filtering low-confidence pairs? We include confidence filtering of pairs as an extra baseline. Our weighting differs from it in one way: it estimates reliability per annotator rather than per item. That should matter only when noise differs across sites, and this is exactly what the uniform-noise versus heterogeneous-noise contrast tests.

Decisive Checks

Three results decide the proposal.

First, the budget curve. If byproduct feedback needs more than about 800 reviewed reports per site to reach the 80% recovery target, the "seconds per report" premise fails at realistic site sizes.

Second, the noise interaction. Reliability weighting must beat uniform weighting under heterogeneous site noise by a margin larger than seed variance across five seeds, and must make little difference under uniform noise. If it helps equally in both conditions, the gain is not coming from handling uneven annotators.

Third, the reader study must show lower time per report for review than for drafting from scratch. Without that, the link to role conflict is rhetorical.

We also report worst-site F1 throughout, because small community hospitals, where residents rotate, are where FedPref showed federation matters most.