The Signal Encoder as an Independent Witness for Report-Based Weak Supervision

Hospitals hold millions of scans and waveforms that come with a free-text report but no clean label. One proven way to label them is cross-modal data programming. Clinicians write a few dozen quick rules over the report text (for example, "report mentions 'hemorrhage' and no negation"). A statistical model combines the rules' noisy votes into probabilistic labels, and an image or signal classifier is trained on those labels. The weak point is the step that combines the votes. It must estimate how accurate each rule is without any ground truth, and it does this only from how the rules agree with each other. Rules written by the same clinician over the same sentences tend to fail together, on negations, hedges, and templated phrasing. Agreement among them is therefore a biased witness.

We propose adding a witness that cannot share those failures: the image or signal itself, read through a self-supervised encoder. A self-supervised encoder is a network pretrained on unlabeled scans or waveforms to predict masked or future parts of its input. If the way a report is written is independent of what the image looks like once the true diagnosis is fixed, then a probe on the frozen encoder becomes a statistically independent source of votes. That holds only if the probe is trained on weak labels from other patients and applied to held-out ones. With such a source, rule accuracies become identifiable without gold labels. They can also be estimated region by region in the encoder's feature space, which shows where text rules break. Of the two directions here, label-efficient pretraining supplies the mechanism and programmatic labeling supplies the problem.

Core Claim

Three conditions together make the target-modality encoder a valid labeling source whose errors are conditionally independent of the report rules' errors. The data must be cross-modal, meaning rules see only the report while the encoder sees only the image or signal. The encoder must be pretrained by self-supervision, so its geometry was never fit to the rules. The probe must be cross-fitted, meaning it is trained on one fold's weak labels and predicts on another. Adding this source to a standard accuracy estimator should give more accurate estimates of rule reliability than agreement-only label models. It should therefore also give better downstream classifiers than the obvious recipe of "Snorkel labels, then fine-tune a foundation model." The claim fails in one testable way. If report errors and image difficulty are correlated, for example when subtle findings are both under-reported and visually faint, independence breaks, and we will measure how much.

Relation to Prior Work

The motivating evidence comes from two places. Cross-Modal Data Programming Enables Rapid Medical Machine Learning has clinicians write roughly 14 report rules per task and trains CNNs from scratch on the aggregated labels. It reports beating physician-months of hand labels by 8.50 ROC-AUC points on average. It never uses the target modality to check the rules; the image model only consumes labels. Advancing Radiograph Representation Learning with Masked Record Modeling pretrains on MIMIC-CXR images and reports jointly and reaches 88.5% mean AUC on CheXpert with 1% labels. This shows paired records yield strong encoders, but it still needs clean labels downstream. We use exactly such an encoder, and to keep the independence argument clean we prefer image-only pretraining. A jointly pretrained encoder may have absorbed report phrasing, and that is one of our ablations.

For feature-aware label combination, we reuse methods rather than compete with their premise. Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision (FABLE) lets rule-correlation patterns vary with fixed instance features through a Gaussian process. Train and You'll Miss It (EPOXY) spreads rule votes to neighbors within a radius in a frozen embedding and tunes that radius on clean validation data. Label Propagation with Weak Supervision propagates votes over a nearest-neighbor graph and is sensitive to graph density. End-to-End Weak Supervision (WeaSEL) learns per-instance rule weights jointly with the classifier from scratch. All four use features to smooth or reweight the rules. None uses features as a source whose independence from the rules is argued and then tested. That independence is what licenses accuracy estimation without gold labels.

The estimator itself comes from Universalizing Weak Supervision. It recovers source accuracies from pairwise agreements under conditional independence, and it flags that this assumption is violated by shared heuristic biases. Our contribution is to construct a source for which the assumption is defensible.

The noise-handling literature constrains the design. Fine-Tuning Pre-trained Language Model with Weak Supervision (COSINE) and Meta Self-Refinement (MSR) show that models trained on weak labels memorize their noise. That rules out using the end model's own in-sample predictions as a "new" source; they would inherit the rules' errors. Cross-fitting is our answer, and in-sample use is our main ablation.

The Estimator

Let $y \in \{-1, +1\}$ be the true label, $\lambda_1, \dots, \lambda_m$ the report rules' votes, and $\lambda_0$ the cross-fitted encoder probe's vote, all in $\{-1, +1\}$. Write $a_i = \mathbb{E}[\lambda_i y]$ for source $i$'s accuracy-like correlation with the truth. If sources $i, j, k$ are conditionally independent given $y$, then $\mathbb{E}[\lambda_i \lambda_j] = a_i a_j$, so

$$|a_i| = \sqrt{\frac{\mathbb{E}[\lambda_i \lambda_j]\,\mathbb{E}[\lambda_i \lambda_k]}{\mathbb{E}[\lambda_j \lambda_k]}},$$

which uses only observable agreement rates (the triplet method). Report rules routinely violate pairwise independence among themselves. We therefore always build triplets using $\lambda_0$ and a decorrelated pair of rule groups. Each group is the merged vote of rules that look at the same report phrases.

The local version repeats this estimate inside each cluster of the frozen encoder's embedding space. That yields rule accuracies conditioned on what the image looks like. Clusters are merged until each holds enough votes for a stable estimate. The output feeds a soft-label model, and the foundation backbone is fine-tuned on those labels as usual.

Research Program

The work has three parts. First, we build the pipeline on MIMIC-CXR, the roughly 370,000 radiograph–report pairs used by MRM. We write report rules for CheXpert-style findings with a fixed time budget matching CMDP's under-8-hour protocol. We pretrain or reuse an image-only masked-autoencoding encoder and train cross-fitted linear probes across K folds.

Second, we test the independence assumption directly on a gold-labeled audit subset. We measure the correlation between probe errors and rule errors given the true label. We report it overall and in a "hard case" stratum: small or subtle findings, and hedged reports.

Third, we replicate on a physiological signal where reports exist. PTB-XL is the candidate if its accompanying report text is usable. The encoder would be a CPC-pretrained state-space model in the style of ECG-CPC from Benchmarking ECG FMs, which outperformed models up to 25 times larger in 5 of 7 task categories. The ECG arm uses PTB-XL's diagnostic labels as the gold reference.

Baselines

The naive reference is majority vote over the report rules, followed by fine-tuning the same pretrained backbone on the resulting labels. It shows what any label model must add.

The strongest weak-supervision competitor is the obvious combination of CMDP and foundation models. Report rules are combined by a standard agreement-only label model, and the same self-supervised backbone is fine-tuned on its soft labels. This is the recipe our method must beat, because it already captures most of the benefit of pretraining. Our method differs only in how the labels are produced. No reported result was found on Lacuna for this combination on MIMIC-CXR or CheXpert, so we run it ourselves. For context, CMDP's from-scratch CNNs beat medium hand-labeled sets by 8.50 ROC-AUC points.

The strongest clean-label competitor is direct fine-tuning of the encoder on gold labels. MRM reports 88.5% mean AUC on CheXpert with 1% labels, and MGCA reports 88.8%, both in the MRM paper. We run it as a learning curve from 50 to several thousand gold labels to locate a crossover point, as in Stronger Than You Think.

The feature-aware competitors are FABLE, EPOXY, and WeaSEL, each given the same frozen encoder features. Where they need clean validation data (EPOXY's radius), they receive the same 100-label validation budget that is also used to tune our method. COSINE-style self-training on the fine-tuned backbone completes the set.

The decisive ablations isolate the key ingredients. The first uses the probe in-sample rather than cross-fitted. The second swaps the self-supervised encoder for a randomly initialized or generic natural-image encoder. The third swaps the image-only encoder for MRM's report-aware encoder, which may leak report phrasing. The fourth runs global rather than cluster-local accuracy estimation.

All methods share data splits, backbone, rule set, fine-tuning schedule, and a matched hyperparameter search. Beating the agreement-only baseline means two things: lower error in estimated rule accuracies against the audit set, and higher mean AUC on the held-out test set across findings, outside seed-to-seed variance. Beating the gold-label curve means our zero-gold pipeline matches at least the 1%-label fine-tuning result.

Why not just…?

Why not just hand-label a hundred images and fine-tune the foundation model? This is the most serious threat. On CheXpert, MRM's 88.5% at 1% labels may already be near the ceiling that weak labels can reach. BOXWRENCH found weak supervision wins mainly on imbalanced, high-cardinality tasks, with crossover at 1,000 to 1,600 clean labels and a persistent 5% gap on Claude9. Rare findings, where random gold samples contain few positives, are where we expect the advantage. We report the crossover point per finding. If our method does not move the crossover beyond a few hundred labels on rare findings, the proposal's practical value is small, and we will say so. We also test the combination BOXWRENCH recommends: initialize from weak labels, then continue fine-tuning on the gold labels.

Why not just use a standard label model and let the foundation model absorb the noise? Strong pretrained encoders may tolerate label noise well; Self-supervised representation learning from 12-lead ECG data reports CPC features are robust to signal noise. That is robustness to input noise, though, not label noise. COSINE and MSR document memorization of systematic heuristic errors in pretrained text models. The agreement-only baseline paired with the same backbone answers this directly. If it matches our method, the extra source is not needed.

Why not just do feature-aware label models like FABLE or EPOXY on the same embeddings? They use features to smooth rule votes but still infer reliability largely from rule agreement. EPOXY additionally needs clean data to choose its radius. If correlated report rules share a failure, smoothing spreads it. The local accuracy estimates on the audit set will show whether FABLE already recovers the same per-region rule failures. If it does, our contribution reduces to an identifiability argument, not a performance gain.

Decisive Checks

The proposal stands or falls on three measurements. The first is conditional error correlation between the cross-fitted probe and the report rules on the audit set; near zero supports the core assumption. The second is whether estimated rule accuracies, global and per cluster, track gold accuracies better than agreement-only estimates. The third is whether this translates into higher CheXpert AUC and a later gold-label crossover, especially on rare findings. A clear negative result on the first check in the hard-case stratum would itself be informative: it would show that report-writing errors and visual subtlety are entangled, which limits every cross-modal labeling scheme, not just ours.