Anatomy-Bounded Classifier Heads for Rare Chest X-Ray Findings

Chest X-ray classifiers for rare findings are mostly trained on labels pulled automatically from radiology reports. Those labels record what a radiologist chose to write down, not what is in the image. Radiologists skip chronic incidental findings, mention some findings mainly when a prior study or an ICU context prompts them, and describe devices whenever the patient has them. A classifier trained with ordinary loss minimization on such labels learns to imitate this reporting behavior. The rare classes, the "tail" of a long-tailed label distribution, are where imitation is cheapest: with a few dozen positives, any contextual cue that happens to co-occur with the report wording is enough to fit the training set. The evidence is already visible. In the CXR-LT challenge, the best teams lost on average 12.3% average precision when scored against expert-annotated images instead of mined labels. Calcification of the aorta was the extreme case, scoring 0.162 against mined labels and 0.688 against human annotation (Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge). In other words, the label source alone can move a class's apparent performance by a factor of four.

We propose to make each rare-finding classifier look only where that finding can physically be. A segmentation model divides the radiograph into anatomical regions such as lungs, mediastinum and aorta, the region under the diaphragm, bones, and soft tissue. Each finding's output head pools features only from the regions where that finding can occur. Pneumoperitoneum reads from the space under the diaphragm, aortic calcification from the aortic region, and rib fracture from bone. The head cannot read cues from anywhere else in the image. We attach this constraint to the decoupled training recipe that already works best for long-tailed chest X-rays: learn features on the natural distribution, then retrain only the classifier with class-balanced sampling. We then test the setting where reporting-habit shortcuts should hurt most, which is the gap between mined-label scores and verified scores and its persistence on prospectively collected studies.

Core Claim

Part of the rare-class failure in long-tailed chest X-ray models is not too few examples. It is the model learning reporting behavior through image context. If that is true, restricting each class head to its anatomical region will do three things. It will lower agreement with mined labels slightly. It will raise agreement with expert-verified labels, and the gain should be largest for tail classes. And it will make predictions insensitive to swapping in image content from outside the class's region. A full-image model with the same decoupled training should show the opposite pattern. The claim fails if the anatomy-bounded heads do no better than the full-image decoupled baseline on verified labels, or if their verified-label gains are matched by simply cleaning the training labels.

This differs from all prior anatomical masking in one respect: the region is chosen per class, not once for the whole image. Earlier work applies a single global mask, either the lung field for CT triage (Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling) or the lesion in breast ultrasound (Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability). On chest X-ray a global lung mask would be actively harmful. Many tail findings, including pneumoperitoneum, aortic calcification, fractures, subcutaneous emphysema and devices, lie outside the lungs. This is likely one reason none of the long-tailed CXR benchmarks have tried masking.

Research Program

Three pieces of prior work shape the design. The first is the motivation for the mechanism. The Role of Pleura and Adipose in Lung Ultrasound AI showed that an unmasked lung ultrasound model graded COVID severity from subcutaneous fat thickness. Masking to the pleura and the region just below it raised AUC from 0.6553 to 0.7742. That was a single global mask on 93 patients, and the paper did not address long-tailed classes. Separately, Risk of Bias in Chest X-ray Foundation Models found that chest X-ray foundation encoders carry sex and race information in their dominant feature directions, with pleural effusion performance dropping 10.7–11.6% for Black patients. This shows that full-image features carry information unrelated to the findings that a flexible head can exploit.

The second piece supplies the training recipe. Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study showed that keeping a frozen backbone and retraining only the classifier on balanced samples (classifier re-training, or cRT) raises NIH-CXR-LT tail accuracy from 1.7% to 30.0%. That study was single-label and full-image. Long-tailed multi-label classification with noisy label of thoracic diseases from chest X-ray contributed two multi-label losses. Adaptive negative regularization stops the flood of negative gradients from swamping rare classes, and large-loss abandoning drops likely-mislabeled examples. We reuse both losses inside the second training stage, which also covers the multi-label case the decoupling study left open.

The third piece constrains the design, in the form of the known weakness of mask cascades: when pathology destroys landmarks, the segmentation fails and the classifier sees corrupted input. We therefore use soft regions. Segmentation probabilities are dilated by a learned margin and pooled as weights, never applied as hard pixel cut-outs. We also log a mask-confidence score on every study, so that we can report performance stratified by segmentation failure. The research file names no chest X-ray anatomy segmentation dataset. We would train or adopt a public multi-structure chest segmentation model and treat its quality as a measured variable rather than an assumption.

Concretely, the system is a shared backbone trained on the natural label distribution of the CXR-LT training set (377,110 images, 26 findings). We extract feature maps once. For each class $c$, the head computes a pooled feature $z_c = \sum_p m_c(p) f(p) / \sum_p m_c(p)$. Here $f(p)$ is the backbone feature at spatial location $p$, and $m_c(p)$ is the soft anatomical weight of location $p$ for class $c$, taken from a fixed finding-to-anatomy map written with a radiologist. The class score is a linear function of $z_c$. Only the heads and the margin of the regions are trained in stage two. The class-to-region map is the one piece of explicit clinical knowledge we add, and it is cheap: one row per finding, written once.

The behavioral side is analyzed directly rather than assumed. For every study, MIMIC-CXR metadata gives the view (portable versus standing), whether devices are present, and whether a prior study exists. These serve as proxies for the reporting context. On adjudicated images we model the probability that a truly present finding was reported as a function of these proxies. This yields a per-class reporting-bias estimate. We then test whether a model's disagreements with the verified labels track that estimate.

Baselines

The naive reference is a standard multi-label network trained end to end on the full image with binary cross-entropy. This is the softmax/ERM baseline whose NIH-CXR-LT tail accuracy is reported at 1.7% in the benchmark study (Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study). It tells us how much of any gain is simply rebalancing.

The strongest existing method on the exact benchmark is the CXR-LT challenge winner. It is an end-to-end, high-resolution backbone with re-weighted loss, reported at test mAP 0.372 and best on 20 of 26 classes (Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge). We will rebuild it on the same backbone, input resolution and data split as our model. Its 0.372 is a reference point; the fair comparison is our reimplementation under an equal tuning budget. The second strong competitor is adaptive negative regularization plus large-loss abandoning, reported at 64.84% total and 66.21% tail balanced accuracy on LTML-MIMIC-CXR against a 56.34% / 56.23% baseline (Long-tailed multi-label classification with noisy label of thoracic diseases from chest X-ray). We will run it on CXR-LT and also on LTML-MIMIC-CXR, so the reported number can be checked against our reimplementation.

The ablation that isolates the key ingredient is full-image decoupled cRT. It uses the same backbone, the same stage-two losses and the same balanced sampling, with heads that pool globally instead of per-anatomy. A second ablation uses a single shared lung mask for every class, to separate "masking at all" from "class-specific masking." A third swaps in a deliberately wrong class-to-region map, which tests whether any spatial restriction helps or only the anatomically correct one does.

All models share one backbone checkpoint where the design allows it. They get identical augmentation, identical stage-two epochs, and the same number of hyperparameter trials, chosen on a held-out split of mined labels only. No method sees the verified labels during tuning.

Beating the strongest baseline means two things together. First, a higher macro-average AP on tail classes against verified labels than both the challenge-style model and full-image cRT, with a paired bootstrap interval that excludes zero. Second, a smaller gap between mined-label and verified-label scores. Matching mined-label mAP is not required, and we expect to lose slightly there.

Why not just…?

Why not just use the challenge winners' end-to-end re-weighted loss at high resolution, since it already wins? It wins on mined labels, and that is the problem: its 12.3% average AP drop on the expert set shows its ranking partly reflects agreement with reporting habits. Our main experiment scores it on verified labels and with context-swap tests. If it shows the same small gap and the same insensitivity to context as our model, the proposal's premise is wrong, and we will say so.

Why not just clean the labels, with large-loss abandoning or a better report parser, instead of constraining the image? This is the most serious threat. If mimicry lives entirely in the labels, label cleaning alone should close the verified-label gap. But label cleaning removes examples a model finds hard. It cannot remove a context cue that predicts the noisy label consistently, and consistent omission of chronic findings looks like signal, not noise. The experiment settles it directly. The full-image cRT model with these noise-filtering losses is one of our baselines, and if it matches the anatomy-bounded heads on verified tail AP, the anatomical constraint adds nothing.

Why not just apply one lung mask, or skip masking and add a loss that penalizes saliency outside annotated regions? A single lung mask removes the evidence for a large share of the tail classes, and the shared-mask ablation will show that loss on extrapulmonary classes. A saliency penalty only discourages reliance on outside regions: the head can still read information the backbone has smeared into the in-region features. Pooling by construction is easier to audit. We will include a saliency-penalty variant at matched budget. If it matches our verified-label and context-swap results, the simpler method should be preferred.

Why not just run a retrospective multi-reader study instead of prospective data? A reader study scores curated images, and these show more textbook cases than live practice does (What benefit can be obtained from magnetic resonance imaging diagnosis with artificial intelligence in prostate cancer compared with clinical assessments?). It also misses how alert volume shapes clinician behavior (Validating Medical Artificial Intelligence in Live Clinical Workflows; Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning). Mimicry shortcuts are exactly the kind of cue that shifts when ordering and documentation practices change over time. A full live deployment like the Hungarian breast rollout (Prospective implementation of AI-assisted screen reading to improve early detection of breast cancer) is beyond this project. We therefore propose a silent prospective phase, described below, and do not claim clinical benefit.

Proposed Experiments

The first experiment is the verified-label comparison. The 406-image CXR-LT expert set is too small for per-class tail estimates, since many tail findings have only a handful of positives there. We will therefore adjudicate an additional enriched set of roughly 1,500 MIMIC-CXR test-split images. The set is sampled so that each tail class has enough candidate positives, with two radiologists plus an arbiter, and CT or follow-up imaging used where available. The primary metric is macro-average AP over tail classes on verified labels. Secondary metrics are head and medium AP and the mined-versus-verified AP gap per class.

The second experiment is the decisive mechanistic check, the context swap. For each test image and class, we replace the image content outside that class's anatomical region with content from another patient while keeping the region intact. We do this with patients matched or mismatched on view type and device presence, then measure how much the predicted probability moves. A model that reads anatomy should barely move. A model that has learned reporting context should move when portable-view or device context is swapped in. We expect full-image models to show large swings on tail classes, and the anatomy-bounded heads to show small swings by design. The informative comparison is between the three ablations: wrong map, shared lung mask, and saliency penalty.

The third experiment links errors to behavior. Within the adjudicated set we regress each model's disagreement with verified labels on the per-study reporting-context proxies. The prediction is that full-image model errors on classes radiologists tend to omit, such as aortic calcification, are explained by those proxies, and that anatomy-bounded errors are not. As a secondary audit we re-run the sex and race subgroup comparison of Risk of Bias in Chest X-ray Foundation Models on CheXpert for pleural effusion. We make no prior claim that the anatomical constraint reduces these gaps.

The fourth experiment is a robustness and prospective check. We stratify all results by segmentation confidence, to measure how much the cascade costs when anatomy is distorted. We then run every model silently on studies from a later time window than the training data and, with a partner site, on several months of newly acquired radiographs. Outputs are hidden from clinicians. Tail-finding outcomes are verified by retrospective adjudication and follow-up imaging. On this stream we report verified tail AP and its change from the retrospective estimate. We also report alerts per 1,000 studies at a fixed sensitivity of 0.8 for each tail class, a direct proxy for alert burden.

The proposal succeeds if the anatomy-bounded heads keep their verified-label advantage on the prospective stream while the full-image models lose more, because that is the retrospective-prospective gap the mimicry hypothesis predicts. It fails cleanly if label cleaning alone or full-image cRT closes the gap.