When a chest X-ray model reports "pneumonia, 0.7," a clinician cannot tell whether the model saw pneumonia or whether it saw pleural effusion and inferred pneumonia because the two often appear together in the training data. Both are legitimate reasons to raise suspicion, but they call for different trust. In standard networks the two are mixed together invisibly inside a shared feature vector. We propose a diagnostic architecture that splits each disease's score into two additive parts. The first is own evidence: image features that a per-disease gate certifies as relevant to that disease, trained so the score cannot be moved by features outside the gate. The second is borrowed evidence: an explicit, sparse, signed term that adds support from the own-evidence scores of clinically related diseases. Because the split is exact by construction, explaining a prediction needs no post-hoc attribution. The borrowing term can also be audited, and it can be turned down or refit when a hospital's disease co-occurrence differs from the training data.
This matters for two reasons the literature names but has not addressed together. Models that lean on co-occurrence predict spurious comorbidities when a disease appears in an atypical context (Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning), and they break when label frequencies shift at test time (Multi-Label Test-Time Adaptation with Bayesian Conditional Priors). Separately, clinicians distrust post-hoc explanations because those explanations may not reflect the computation that actually produced the score. A model whose dependency use is a separate, visible term addresses both problems with the same piece of structure.
Core Claim
Making label dependency an explicit additive channel on top of enforced label-specific evidence does three things. It matches the accuracy of entangled dependency models such as FACT. It reduces false positives on atypical cases, meaning images where a disease's usual companion is absent. It also gives explanations that are faithful by construction. The essential ingredient is enforcement. If each disease's own-evidence head may use any feature, the network quietly absorbs comorbidity into "own" evidence, and the decomposition becomes cosmetic. The core claim therefore rests on the claim that enforcement changes behavior, and we test that directly.
How This Differs from Existing Work
Three papers come closest, and each supplies one piece while leaving out the combination.
Dual Perspective of Label-Specific Feature Learning for Multi-Label Classification (DELA) supplies the enforcement mechanism. For each label, DELA learns a gate that marks which shared feature dimensions are irrelevant, injects random noise into those dimensions, and trains the label's classifier to be unaffected by the noise. DELA learns each label's gate independently and explicitly leaves label dependencies as open work. It was also evaluated only on generic benchmarks such as Corel5k and Mediamill, not on medical images.
FACT motivates the clinical setting. It shows that comorbidity structure and "visual polysemy" (one visual sign supporting several diseases) matter, and it halves calibration error on ODIR-5K by measuring distances between shared vector-quantized visual atoms and anchors propagated over a comorbidity graph. In FACT, however, dependency is embedded in the anchors themselves. It cannot be separated from visual evidence or switched off, and the file notes that FACT is sensitive to errors in its fixed co-occurrence prior.
SEMGNN is the closest in spirit. It generates label-conditioned explanation masks that share evidence among correlated labels and trains them for sufficiency and necessity. It works only on graph edges, and it lets correlation shape the masks themselves. That is the reverse of our design: we keep masks pure and route dependency through a separate term.
Interpretability-by-architecture in imaging motivates the overall approach. PN-SAMP forces a network to predict radiological attributes alongside malignancy. The ChestX-Ray8 work derives localization heatmaps from image-level labels. Integrating Clinically-Relevant Features into Skin Lesion Classification adds a caution: when a shared backbone is forced to predict human-defined criteria, accuracy on benign nevi drops. We read that result as a warning against shared-representation constraints. Per-label gates avoid that particular failure, because one label's constraint does not reshape another label's features.
Finally, Generalization Analysis for Label-Specific Representation Learning shows that decoupled label-specific representations have error bounds that do not grow with the number of labels. It also shows that bounds for graph-convolution label models worsen with node degree and network depth. This constrains our design. We keep dependency as a shallow, sparse, linear term rather than a deep graph network, so most of the model stays in the decoupled regime.
Model
A backbone (a DenseNet-121 or ResNet-50, matching prior chest X-ray and FACT work) produces a spatial feature map $h(x)$. Each disease $k$ has a gate $g_k$ over both spatial positions and channels, learned with the Gumbel-Softmax relaxation (a differentiable approximation to sampling discrete on/off choices), as in DELA. A small per-disease scorer $s_k$ reads only the gated features. The final logit is
$$z_k(x) = s_k\big(g_k \odot h(x)\big) + \sum_{j \in N(k)} w_{kj}\, s_j\big(g_j \odot h(x)\big),$$
where $\odot$ is elementwise masking, $N(k)$ is the set of neighbors of disease $k$ in a comorbidity graph built from training-set co-occurrence (edges kept for strong positive or negative association), and $w_{kj}$ are signed weights trained with an L1 penalty so that most of them become exactly zero. Signed weights let the model express mutual exclusion as well as co-occurrence.
The invariance loss injects Gaussian noise into the features where $g_k$ is off and penalizes any change in $s_k$. Shared dimensions are allowed: if a feature is gated on for two diseases, it counts as own evidence for both, which respects visual polysemy. What the loss forbids is disease $k$ silently using features that its own gate marks as irrelevant.
Each prediction is then explained by a map, namely the gate $g_k$ weighted by its contribution to $s_k$, together with a short ledger of borrowed terms such as "+0.4 from effusion evidence." The ledger is exact because the decomposition is additive.
A natural objection is that the borrowing term is only stacking, meaning a second-stage classifier that reads the outputs of the first-stage per-label classifiers. Structurally, it is, and that is deliberate. Our contribution is that the inputs to the stacking are certified own-evidence scores rather than per-label outputs that already contain hidden comorbidity. We also test a richer, feature-level borrowing variant. We keep it only if it helps.
Research Program
The program has four parts.
The first is building and training the model on NIH ChestX-ray14 (14 pathologies, about 112k images) and ODIR-5K (8 ocular categories). For ODIR-5K we use the splits and medical priors released with the FACT codebase, so comparisons with FACT are made on identical data.
The second is constructing atypical-case subsets from the test labels alone. These are images positive for disease $k$ but negative for its strongest comorbid partner, and images negative for $k$ but positive for that partner. They isolate co-occurrence-driven errors without any new annotation.
The third is building controlled shift test sets by resampling the NIH test set so that chosen disease pairs become independent or anti-correlated. We then evaluate three versions of our model: as trained, with the borrowing term removed, and with the weights $w$ refit on a small labeled sample from the shifted set. The refit involves only a few dozen parameters.
The fourth is grounding the own-evidence maps against bounding boxes from ChestX-Det10, a 542-image test subset of NIH with boxes for 10 abnormalities. We use the intersection-over-detected-box localization protocol of ChestX-Ray8, in which a map counts as a hit if its overlap with the expert box passes a threshold.
Baselines
The naive reference is a DenseNet-121 trained with independent binary cross-entropy per disease. This is the CheXNet recipe, reported at an AUC-ROC of 0.8413 on ChestX-ray14 in Chest X-rays Classification: A Multi-Label and Fine-Grained Problem. It sets the floor for both accuracy and spurious-comorbidity behavior.
That same paper's dependency-aware multi-label softmax loss with dual-stream bilinear pooling reports an AUC-ROC of 0.8537. It is the strongest reported chest X-ray number in the file. Splits differ across papers, so we retrain it on our split rather than quoting it.
The strongest existing method on our question is FACT. It models comorbidity explicitly for medical images and reports ODIR calibration error falling from 27.51% to 16.22%, along with a +2.92% gain in mean average precision (mAP). The file gives only the gain, so we report absolute mAP from our own reruns. We run FACT from its official code, on the same splits and backbone, with the same hyperparameter search budget as our model (an equal number of trials and equal GPU hours).
Beating FACT means matching its overall mAP within one standard deviation across seeds while doing at least one of the following:
- Clearly lowering the false-positive rate on the atypical-case subsets.
- Degrading less under resampled co-occurrence shift.
Two ablations isolate the key ingredient. The first, the unenforced ablation, keeps the identical architecture and additive borrowing but removes the gates and the noise-invariance loss, so own-evidence heads can see all features. The second, the no-borrowing ablation, keeps the gates and invariance loss but sets $w = 0`; in other words, it is DELA adapted to spatial medical features.
For localization, we compare against Grad-CAM applied to each baseline and against the ChestX-Ray8 pooling heatmaps, which report 0.99 localization accuracy for cardiomegaly at an overlap threshold of 0.1. Faster R-CNN (40.2 mAP on ChestX-Det10) is box-supervised. We cite it as a ceiling, not a competitor.
Why not just…?
Why not just train per-disease attention heads on a shared backbone and stack a co-occurrence model on top? This is the real threat, and it is precisely our unenforced ablation. Its weakness is that, without the invariance constraint, each head is free to look at the companion disease's region, so the "own" score already contains comorbidity, and the stacked term understates how much the model depends on it. The experiment settles this directly. If the unenforced ablation matches our model on atypical-case false positives and on shifted test sets, then enforcement is unnecessary, and we would report the simpler stacked model as the contribution.
Why not just apply Grad-CAM or SHAP to a strong model such as FACT? Post-hoc maps show where a model looked, not whether that evidence belonged to the predicted disease. In atypical cases we will measure how often the pneumonia Grad-CAM map falls inside effusion boxes on ChestX-Det10. SEMGNN's synthetic results show that a post-hoc-style explainer recovers far fewer true rationales than an intrinsic one (ground-truth explanation F1 of 0.311 versus 0.463), and DILA reports that KernelSHAP can take over an hour per clinical instance. Neither result is on images, so our localization comparison is the evidence that counts here.
Why not just use FACT, or a graph convolution network over labels, and accept that dependency is entangled? If entangled dependency were harmless, FACT would hold up under shift and on atypical cases as well as our model does. The resampled-shift test and the atypical subsets measure exactly this. The theoretical result that label-graph models degrade with node degree, and the documented sensitivity of FACT to a noisy prior, predict that it will not hold up. We treat that as a hypothesis to test, not a conclusion.
Decisive Checks
The idea succeeds only if three things hold together, across five seeds on both datasets:
- Overall AUC and mAP stay within noise of FACT.
- On atypical-case subsets, our model's false-positive rate is lower than both FACT's and the unenforced ablation's.
- Removing or refitting the borrowing term recovers more performance under resampled co-occurrence shift than the baselines can.
A fourth check tests whether "own" means something visual. On ChestX-Det10, own-evidence maps must localize better than Grad-CAM for the diseases whose borrowed share is large. If enforcement buys nothing over the unenforced ablation, the central claim fails, whatever the headline accuracy.
References
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning
- Multi-Label Test-Time Adaptation with Bayesian Conditional Priors
- Dual Perspective of Label-Specific Feature Learning for Multi-Label Classification
- FACT: Fuzzy Alignment with Comorbidity Topology for Reliable Multi-Label Medical Image Diagnosis
- Self-Explainable Multi-Label Graph Neural Network for Correlated Evidence Attribution
- Joint learning for pulmonary nodule segmentation, attributes and malignancy prediction
- ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
- Integrating Clinically-Relevant Features into Skin Lesion Classification
- Generalization Analysis for Label-Specific Representation Learning
- ChestX-Det10: Chest X-ray Dataset on Detection of Thoracic Abnormalities
- Chest X-rays Classification: A Multi-Label and Fine-Grained Problem
- DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction