Region-Anchored Concept Graphs for Rare Chest X-ray Findings

Rare chest X-ray findings are hard to learn and hard to explain for the same reason. Each one leaves a small, spatially specific trace, and those traces often appear in anatomically distant places. A pneumothorax can show as a thin pleural line at the apex together with a deepened costophrenic angle at the base. Pulmonary edema shows up as matched opacities in both lower lungs. We propose a concept bottleneck model, meaning a classifier forced to predict through a layer of human-readable intermediate findings. In our version, every intermediate concept is a finding tied to a specific anatomical region, such as "opacity, left lower zone." These region-concepts share evidence over two kinds of links: links between physically adjacent regions, and a small number of learned "functional" links between distant regions that tend to show the same pathology. The idea borrows from urban traffic forecasting, where a downtown and a distant suburban business district behave alike while the residential area between them does not. The final disease predictions are a sparse linear readout of these region-concepts, and only that readout is retrained under class-balanced sampling to correct for the long tail.

The payoff we expect is twofold. First, a rare disease with a few hundred examples can borrow evidence from functionally linked regions, much as a data-poor city borrows from a data-rich one. Second, every diagnosis decomposes into named regional findings that can be checked against radiologist segmentations. Post-hoc heatmaps demonstrably cannot provide that check. The claim is falsifiable: if the functional links are doing real work, the gains on tail diseases should be concentrated in findings whose evidence is spatially distributed, and they should disappear when the links are randomly rewired.

Core Claim

Our central claim concerns what the concept nodes are. A concept bottleneck whose nodes are region-grounded findings, and which propagates evidence over a sparse two-type graph (physical adjacency plus learned non-local functional links), can reach tail-class detection close to a rebalanced black-box model on long-tailed, multi-label chest X-ray benchmarks. It does so while producing explanations that localize pathology better than gradient saliency. Applying the rebalancing step only to the interpretable concept-to-disease readout is what makes the long-tail correction itself auditable.

The closest prior work does part of this. Graph Concept Bottleneck Models learn a latent graph among concepts and pass messages over it, improving accuracy by 1–2% over non-graph bottlenecks (77.95% vs 74.69% on CUB). Its concepts, however, are abstract semantic tags with no spatial location, and it does nothing about class imbalance. FACT runs a graph network over a disease co-occurrence graph and evaluates on the CXR-LT long tail. Its graph lives purely at the level of global labels, is fixed from co-occurrence statistics that are noisy for rare diseases, and exposes no concepts a clinician can intervene on. LTCXNet captures label dependencies with ML-Decoder cross-attention and a three-model head/tail/all ensemble, and remains opaque. Our proposal differs from all three in the unit of representation, the (region, finding) pair, and in where the graph's edges come from: anatomy plus learned long-range functional similarity, not label co-occurrence.

Research Program

The model has three stages. The first stage is a backbone image encoder whose feature map is pooled into a fixed set of anatomical regions. These are roughly a dozen zones: upper, middle, and lower lung zones on each side, the costophrenic angles, the cardiac silhouette, the mediastinum and aortic knob, and the subdiaphragmatic area. The regions come from an off-the-shelf lung and heart segmenter plus fixed geometric rules. No Lacuna resource was found for this component, so we treat it as a replaceable preprocessing step and report sensitivity to it.

The second stage uses a small per-concept head to score each (region, finding) pair, giving an initial concept score $h_{rc}$ for region $r$ and finding type $c$. These scores are then propagated once:

$$\tilde h = h + A_{\text{phys}} h + A_{\text{func}} h,$$

Here $A_{\text{phys}}$ is a fixed, row-normalized adjacency over neighboring regions (same finding type). $A_{\text{func}}$ is a learned, top-$k$ sparse, and deliberately non-symmetric matrix that may connect any two region-concepts, including distant regions and different finding types. Allowing asymmetry lets "enlarged cardiac silhouette" push evidence toward "basal effusion" without the reverse link being equally strong. Keeping $A_{\text{func}}$ sparse and applying a single propagation step is a direct response to the over-smoothing risk noted for dense non-local graphs in the traffic literature.

The third stage predicts each disease $k$ as $y_k = \sigma(\sum_{r,c} w_{k,rc}\,\tilde h_{rc} + b_k)$, where $\sigma$ is the logistic function and an L1 penalty keeps the weights $w_{k,rc}$ sparse. A clinician can then read "left-lower opacity plus right-lower opacity plus cardiac enlargement" off the weights for edema.

Supervision for the region-concepts comes from report text. Following the approach of ProtoSR, which used an instruction-tuned language model to mine more than 227,000 MIMIC-CXR reports into structured findings with anatomic location, we extract (finding, location) mentions and map them onto our regions. This resembles how Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models used GPT-4 to define chest X-ray concepts. That work, however, kept a global, independent, linear concept layer on binary tasks. We ground concepts spatially and couple them.

For training we follow the finding of Long-Tailed Classification of Thorax Diseases on Chest X-Ray that the long-tail failure lives mostly in the classifier head. Stage one trains everything end to end on the natural distribution. Stage two freezes the encoder, the concept heads, and the graph, and retrains only the sparse readout $w$ under class-aware sampling. We add the adaptive negative-logit regularization of ANR/LLR, which counters the flood of negative gradients that multi-label binary losses produce for rare diseases.

Because the readout is sparse and linear, we can report exactly which regional findings rebalancing up-weighted for each tail disease. We will also check that these changes are clinically plausible rather than shortcuts.

Baselines

The naive reference is a ConvNeXt or DenseNet trained with plain binary cross-entropy on the natural distribution, with Grad-CAM for explanations. It anchors how badly tail classes fail by default: Holste et al. report 1.7% softmax tail accuracy on NIH-CXR-LT.

The strongest performance competitor is the same backbone with decoupled classifier retraining (cRT) plus ANR. Holste et al. report 30.0% tail accuracy on NIH-CXR-LT and 36.7% on MIMIC-CXR-LT for cRT, and ANR-LLA reaches 64.84% total and 66.21% tail balanced accuracy on LTML-MIMIC-CXR. For the multi-label challenge setting, the reference is the CXR-LT top team's test mAP of 0.372 from the CXR-LT challenge overview. We will also run LTCXNet's ML-Decoder head, because its label queries attend across the whole feature map and so already model something non-local. No reported numeric result for FACT on CXR-LT was found on Lacuna (only "dominant lead" on tail diseases), so we rerun it ourselves rather than quote it.

The strongest interpretable competitor is Graph CBM with semantic-only concepts (findings without regions), given the same backbone, the same two-stage rebalancing, and the same report-mined supervision collapsed across regions. No reported number on ChestXpert was found on Lacuna for Graph CBM, so it is rerun as well.

The ablations that isolate our key ingredients are the same region-concept model with four graph variants: no graph, adjacency only, a dense all-pairs learned graph, and a degree-matched random rewiring of $A_{\text{func}}$. A further ablation rebalances the whole network rather than only the readout.

For fairness, all methods share one backbone checkpoint family, input resolution, augmentation (no MixUp), training epochs, and a fixed hyperparameter search budget of the same number of trials per method, with selection done on the validation split.

We will count the proposal as beating the field if it meets three conditions. Its tail-group mAP on CXR-LT must be within one point of the best black-box competitor, and it must exceed semantic Graph CBM on tail mAP. Its localization on CheXlocalize must clearly exceed Grad-CAM on the black box. Matching the black box on head classes is not required.

Why not just…?

Why not just train a rebalanced black box and explain it with Grad-CAM? This is the real performance threat, and we expect it may win on head-class mAP. The explanation half, however, is measurably weak. Benchmarking saliency methods for chest X-ray interpretation found saliency 24.0% worse than radiologists in mIoU and 76.2% worse on lung lesions. Unlearning Spurious Correlations in Chest X-ray Classification shows that a model reaching 0.95 accuracy can be relying on bone epiphyses. Our region-concept activations give a localization map by construction, and we score both on CheXlocalize. If the black box's tail mAP exceeds ours by more than a point, we will report that as the honest price of interpretability.

Why not just use Graph CBM with rebalancing added? This tests whether spatial grounding matters at all. Our prediction is specific: semantic concepts cannot tell "opacity in both bases" from "opacity at one base," so the advantage of region-concepts should appear on tail diseases whose evidence is spread across regions, and should vanish on focal ones such as nodules. If the gain is uniform, or absent, the spatial story is wrong.

Why not just let a transformer or ML-Decoder attend over all patches, which is already non-local? Dense attention can learn any long-range link, and LTCXNet's tail gains (79% on pneumoperitoneum) show it helps. Its attention is not constrained to named anatomy, however, and dense aggregation risks head classes swamping tail signals. The dense-graph ablation inside our own model addresses exactly this. If dense links match sparse functional links on tail mAP and localization, we will drop the sparsity claim.

Why not just skip the graph and use a region-level bottleneck alone? It is possible that grounding alone does all the work. The no-graph and random-rewiring ablations settle this. The functional links are justified only if they beat degree-matched random edges on tail classes.

Proposed Experiments

We train on CXR-LT (26 findings, 377,110 images) and report per-group mAP (head, medium, tail). Because text-mined labels overstate performance by an average 12.3% AP according to the CXR-LT challenge, we give primary weight to its 406-image radiologist gold-standard test set. We also report MIMIC-CXR-LT tail accuracy for direct comparison with cRT, and balanced accuracy on LTML-MIMIC-CXR's 26 rare categories for comparison with ANR-LLA.

To probe the open problem of memorizing training co-occurrences, we evaluate the MIMIC-trained model on the overlapping classes of NIH-CXR-LT, whose disease prevalences differ. We measure whether the learned functional links transfer better than FACT's fixed co-occurrence graph.

For interpretability, we compute mIoU and hit rate on CheXlocalize from the region-concept activations and compare them with Grad-CAM and Integrated Gradients on the black-box baselines. We also run test-time intervention: we correct a random fraction of region-concepts to their report-derived values and measure how much disease mAP improves, relative to Graph CBM.

Three decisive checks determine whether the idea stands. The first is the random-rewiring control: functional links must beat degree-matched random links on tail mAP. The second is the pre-registered split of tail diseases into spatially distributed and focal findings: gains must concentrate in the distributed group. The third is the readout audit: after rebalancing, the up-weighted region-concepts for each tail disease must be judged anatomically plausible by a radiologist blinded to the method. Failing the first two would mean non-local spatial structure is not the mechanism. Failing the third would mean rebalancing is merely amplifying shortcuts.