When a hospital wants a multi-label diagnostic model, meaning one that flags every finding present on a scan rather than picking a single disease, it usually has only a few dozen expert-labeled studies for its own population. With that little data, the model cannot learn which findings tend to appear together. It also cannot learn which findings rarely appear together, or which findings become likely once another is present. Large public chest X-ray corpora carry exactly this information, but in a corrupted form. Their labels were mined automatically from radiology reports, and radiologists often leave secondary findings out of a report once they have described the main one. This proposal takes the disease dependency structure from those large noisy corpora, removes the specific bias that report omission introduces, and transfers only the part of that structure that should stay stable across hospitals. A small classifier trained on the local expert labels then supplies the rest.
The opportunity lies in a gap between two literatures. Methods that model disease dependencies, such as FACT, which builds a comorbidity graph and aligns image features to it, assume the graph comes from a fully labeled multi-center cohort. Few-shot and semi-supervised chest X-ray methods, such as Self-Training with Improved Regularization, learn without any dependency model, and that paper reports that its models degrade when disease prevalence shifts. The question that matters in practice sits between them. Where should a dependency prior come from when the target cohort is too small to estimate one? Which part of it survives the move from a large, biased source to a small, shifted target?
Core Claim
We claim that the pairwise dependency between two findings can be estimated reliably from report-mined labels, provided two conditions hold. First, the dependency is expressed as a log odds ratio, which measures how much one finding raises or lowers the odds of another and does not depend on how common each finding is. Second, the estimate is corrected for selective omission using the target's own small expert-labeled set. Under these conditions the transferred dependency improves few-shot multi-label diagnosis in two ways. It beats a dependency estimated from the target cohort alone. It also beats the dependency implicitly absorbed by pretraining on the same noisy labels. We expect the gains to concentrate where they matter most clinically: at very small target sizes, under prevalence shift, and on rare findings that co-occur with common ones.
The model is deliberately simple, a pairwise Markov random field over labels. For an image $x$ and a binary label vector $y \in \{0,1\}^L$ over $L$ findings, it scores label configurations as
$$ p(y \mid x) \propto \exp\Big(\sum_{i} y_i\, f_i(x) + \sum_{i<j} W_{ij}\, y_i y_j\Big). $$
Here $f_i(x)$ is the per-finding score from an image classifier fit on the target set, and $W_{ij}$ is the pairwise coupling between findings $i$ and $j$. When the population shifts only in how common each disease is, a shift absorbed by the per-finding terms, the couplings $W$ need not change. This is why we transfer $W$ from the source and refit everything else on the target. The structure has the same order of magnitude of parameters as the label space squared, so it adds no graph network on top of the backbone. That is a direct response to the overhead that FACT and PCLP report. It also needs no arbitrary label ordering, which PCLP's chain-rule causal decomposition requires.
Research Program
The work has three parts. The first is estimating source couplings from report-mined labels. Following the reporting-bias diagnosis in AGXNet, we model each mined label as the true label multiplied by a probability that the radiologist mentions it. That mention probability may depend on which other findings are present. Concretely, a secondary finding $j$ is mentioned with probability $r_j$ when no other major finding is present, and with a lower probability $r_j'$ when a dominant finding is present. Fitting the coupled model to mined labels without this correction biases $W$, typically toward spurious negative dependence between a dominant finding and its frequent companions.
The second part, which we consider the most useful piece, uses the small target expert set twice. Its images train $f$. Its complete expert labels also give an audit sample for estimating the mention probabilities, provided the matching reports are available, or a source-side audited subset is used otherwise. Annotation is therefore not spent on two separate tasks. When a pair has little evidence, we shrink its coupling toward zero, with the shrinkage strength set by the effective sample size behind that pair. This answers the critique in Correlative and Discriminative Label Grouping that over-weighting raw co-occurrence makes classifiers hallucinate secondary labels. In our model a weakly supported coupling simply does nothing.
The third part combines the two at training and inference time. The unary classifier is a pretrained chest X-ray backbone with per-finding heads, fine-tuned on the $K$ target studies. At inference we compute exact or mean-field marginals of the joint model. The same marginals also serve as structured pseudo-labels for the target's unlabeled images, in the spirit of PCLP's argument that pseudo-labels ignoring label dependence cannot recover the true posterior. Unlike PCLP, our prior is imported from a different, larger population and is not learned adversarially from the small one. Spatial localization stays out of scope. The method is about image-level diagnosis, where the dependency question can be tested cleanly.
Proposed Experiments
The source corpora are ChestX-ray14, introduced with ChestX-Ray8, whose 14 labels were mined from reports with a reported NLP F1 of 0.90, and MIMIC-CXR. The target is VinDr-CXR, whose radiologist-annotated boxes for 14 diseases give clean image-level labels on overlapping findings. We simulate few-shot cohorts by sampling $K \in \{10, 25, 50, 100\}$ target studies over many random draws. We also build prevalence-shifted splits by resampling target test sets so that marginal disease rates differ from the source. A second target, held for robustness, is the expert-boxed subset of ChestX-ray14 when MIMIC-CXR is the source.
The metrics are mean AUC and mAP per finding, reported separately for rare findings. We add a hallucination rate, defined as the false-positive rate on finding $j$ among test images where a strongly coupled finding $i$ is present and $j$ is absent, and a calibration error in the style FACT reports. All variants share the same backbone, the same target samples, and the same hyperparameter search budget.
Baselines
The naive reference is binary relevance: the shared backbone pretrained on source mined labels, fine-tuned on the $K$ target studies with independent per-finding heads and weighted cross-entropy. This is the setup in which ChestX-Ray8 found weighting helps but rare diffuse findings stay weak, for example Pneumonia AUC rising from 0.51 to 0.63. It is also the strongest simple competitor, because source pretraining already exposes the backbone to co-occurrence patterns. Beating it means a consistent AUC and mAP gain across draws at $K \le 50$, together with a lower hallucination rate.
The strongest semi-supervised competitor is the self-training recipe of Self-Training with Improved Regularization, which combines mixup, confidence tempering, and a noisy student. It reports W-AUC 0.844 and W-PRC 0.684 on five CheXpert findings with 12.5k labeled images. That regime is far larger than ours, and we found no reported result on Lacuna for any method on few-shot VinDr-CXR. We will run this recipe on our splits with the same unlabeled pool, both alone and with our joint marginals substituted as its pseudo-labels. For dependency-aware competitors, we run a PCLP-style causal prior learned from the target set, which reports 77.25% mAP against 65.40% for plain cross-entropy on PASCAL VOC with 5% labels. We also run a FACT-style graph-convolution comorbidity model whose graph is built from raw source co-occurrence, and the Label-Assemble recipe of training on assembled partial labels, which reports mean AUC 0.832 on ChestX-ray14 at full scale.
The ablations isolate the key ingredient. They are: couplings set to zero; couplings estimated only from the $K$ target studies; couplings from raw uncorrected mined labels; couplings transferred as full joint co-occurrence frequencies rather than prevalence-free odds ratios; and the full model with shrinkage removed. The central claim holds only if the corrected odds-ratio couplings beat all of these, and only if the gap over joint-frequency transfer widens on the prevalence-shifted splits.
Why not just…?
Why not just pretrain on the large noisy corpus and fine-tune? This is the real threat. The backbone's shared features may already encode which findings travel together, leaving nothing for an explicit prior to add. Label-Assemble's finding that related diseases help rare targets through a shared backbone, without any explicit graph, points in this direction. Our binary relevance baseline is exactly this pipeline, so the experiment settles the question directly. If explicit couplings give no gain over it at any $K$, or on the shifted splits, the claim fails and we will report that.
Why not just estimate co-occurrence from the target cohort itself? With 10 to 50 studies, most finding pairs co-occur zero or one times. A target-only estimate is mostly noise, and Correlative and Discriminative Label Grouping shows that trusting such coincidences produces hallucinated labels. The target-only ablation measures this cost, and we expect it to shrink as $K$ grows. Locating the crossover point is itself a useful result for practitioners.
Why not just use the mined co-occurrences as they are, as FACT-style graphs do? FACT itself notes that comorbidity graphs built from noisy clinical text contaminate its semantic anchors. AGXNet documents the reporting bias in which radiologists omit secondary findings once the main one is described. The raw-coupling ablation tests whether the omission correction matters. If raw and corrected couplings perform the same, the correction is unnecessary and the method becomes simpler, which we would also report.
Why not just use a frozen vision-language model zero-shot? RAM shows that frozen CLIP-style models produce false positives in multi-label scenes and need dedicated adapters. HARP shows that medical vision-language representations scatter a single finding across more than 70 latent features. These results suggest such models are a weak unary scorer rather than a substitute for dependency modeling. Our couplings can still sit on top of any such scorer, and we will include a zero-shot unary variant to check this.
Decisive Checks
Three results would establish the claim. The first is a consistent gain of corrected odds-ratio transfer over source-pretrained binary relevance at $K \le 50$ on VinDr-CXR. The second is a gap over joint-frequency transfer that widens under prevalence shift. The third is a lower hallucination rate than raw-graph and target-only couplings. Any one of these alone would be weak evidence. If the first fails, the proposal reduces to the statement that pretraining already captures disease dependency, which is a useful negative result, and we would publish it as such.
References
- FACT: Fuzzy Alignment with Comorbidity Topology for Reliable Multi-Label Medical Image Diagnosis
- Self-Training with Improved Regularization for Few-Shot Chest X-Ray Classification
- Correlation-Induced Label Prior for Semi-Supervised Multi-Label Learning
- Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning
- ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
- Label-Assemble: Leveraging Multiple Datasets with Partial Labels
- Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transport
- Auditing Clinical Concept Fragmentation in Sparse Medical Vision–Language Representations