Anatomy-Relative Geometry as a Faithful Bottleneck for Chest X-ray Findings

Radiologists rarely call a chest X-ray finding from its appearance alone. A pneumothorax, where air collects between the lung and the chest wall, is a lucent band whose position relative to the chest wall is what defines it. A pleural effusion is fluid that blunts the angle at the bottom of the lung. Cardiomegaly is a heart that is too wide relative to the chest. Current diagnostic networks map the whole image to a label. They then justify that label with heat maps, and those heat maps have been shown to depend on pixels outside the region they highlight. We propose a detector re-scoring model whose only path from pixels to a decision runs through two narrow channels. The first is appearance features computed on a hard crop of the candidate finding. The second is a short vector of relative geometric measurements between that finding and segmented anatomy (lungs and heart). Each decision can then be stated as a measurable sentence, such as "opaque region, 80% inside the lower third of the left lung, touching the lateral lung border." The model cannot secretly use a hospital tag or a scanner artifact elsewhere in the image, except through the anatomy segmenter, which we audit separately.

The opportunity comes from two developments. In natural-image relation recognition, small hand-defined vectors of relative offsets, scale ratios and overlaps carry much of the signal. Separately, large public anatomical mask collections now exist for chest radiographs. Nobody has joined the two to test a specific, falsifiable prediction. If geometry relative to anatomy is what defines a finding, then adding it should help most on exactly those findings (pneumothorax, effusion, cardiomegaly, aortic enlargement). It should help little on findings defined mainly by texture (nodules, calcification). It should also make the model more robust when it moves to a new hospital.

Core Claim

Take a standard chest X-ray detector's candidate boxes. Re-score each one with a model built from a crop-restricted appearance term and an anatomy-relative geometry term. Our claim is that this re-scorer matches or beats the detector's own scores for anatomy-defined findings, with gains concentrated in those classes. Its predictions also stay invariant, by construction, to image edits outside the crop and the anatomy masks. Finally, it degrades less than end-to-end scoring under cross-hospital transfer. The novelty is not "use geometry in medical imaging" in general. It is the specific decomposition, the per-class prediction of where gains should and should not appear, and a faithfulness test adapted from the prototype-misalignment literature that end-to-end explanations fail and this model passes.

Relation to Prior Work

Three lines of work motivate the relative encoding.

  • LinkNet adds a layout module that encodes relative box coordinates, scale and aspect ratios to predict scene-graph predicates on Visual Genome.
  • Ramisa et al. use an 11-dimensional vector of offsets, scale ratios and overlap to predict spatial prepositions. They find vertical offset to be the single most important feature.
  • Hosang et al. show that a network can learn to suppress duplicate detections from box geometry alone, without pixels.

We borrow the descriptor design from the first two and the "re-score detections from geometry" framing from the third. We differ in two ways. The reference entities here are anatomical structures rather than other detected objects. The measurements are computed on masks, because diffuse findings are poorly served by boxes, as Human-centric Relation Segmentation argues for amorphous natural-image regions.

Three medical systems supply the bottleneck idea but stop short of relative geometry.

  • Pisov et al. replace midline-shift regression with an explicit one-coordinate-per-row midline predictor. This cuts error nearly in half, but the constraint describes a single structure.
  • Yang et al. use lesion detections as a weighting map for grading retinal disease. They never measure where lesions lie relative to retinal anatomy. - Lesion-Centric Latent Phenotypes pools encoder features inside a predicted breast lesion mask and arbitrates single-lesion shape descriptors. It has no lesion-to-anatomy measurements.

Our crop-restricted appearance channel is a stricter version of that paper's mask-weighted pooling. Masking deep features does not stop the receptive field from seeing outside the mask; cropping pixels does.

One paper constrains the design. The spatial misalignment benchmark shows that prototype models' explanations move when pixels outside the explained region are perturbed. Its alignment loss reduces this but does not eliminate it, and costs accuracy and roughly 40% training time. This is why we enforce locality architecturally rather than through a loss. It also gives us a ready-made metric family to adapt.

Medical scene-graph resources set the gap. Chest ImaGenome, RadGraph and CXR-QBA encode "located at" as a discrete symbolic edge. None use continuous relative measurements to drive a diagnostic decision.

Research Program

Inputs. For each radiograph we take candidate boxes from a Faster R-CNN detector and lung and heart masks from CheXmask. CheXmask covers both VinDr-CXR and the NIH images underlying ChestX-Det10. Its per-image reverse-classification-accuracy quality estimate lets us exclude masks below its recommended threshold, an estimated Dice of 0.7.

Relative descriptor. For candidate region $F$ and each anatomical structure $A$, we compute a descriptor $\psi(F,A)$ with the following entries:

  • centroid offsets, normalized by the lung-field width and height;
  • the log area ratio;
  • the fraction of $F$ lying inside $A$;
  • the signed distance from $F$'s centroid to $A$'s boundary, normalized by lung height;
  • the vertical position of $F$ as a fraction of $A$'s craniocaudal extent;
  • a left/right indicator;
  • the cardiothoracic ratio of the image.

Where a finding is diffuse, we refine the box to a coarse region by thresholding the detector's feature map inside the box. We report box-only and refined variants separately.

Scoring model. The re-scored logit for class $c$ is

$$s_c(F) = g_c\big(\phi(\text{crop}(I,F))\big) + h_c\big(\psi(F,A_1),\dots,\psi(F,A_K),\,p\big),$$

where the symbols are:

  • $I$ is the image;
  • $\phi$ is a small CNN applied only to the pixels of the enlarged crop around $F$;
  • $g_c$ and $h_c$ are small networks;
  • $A_1,\dots,A_K$ are the anatomical masks;
  • $p$ is an indicator for anteroposterior versus posteroanterior projection. We include it because the anteroposterior view magnifies the heart, which distorts relative size in a known direction.

The additive form means each prediction splits exactly into an appearance share and a geometry share. Because $h_c$ sees only a handful of named measurements, its dependence can be plotted directly as partial-dependence curves and rendered as a sentence.

Faithfulness. The guarantee is conditional, and we state it that way. Pixels outside the crop can influence the score only by changing the anatomy masks. Any such change is visible, because the masks are part of the explanation. We measure how often it happens.

Discordance cases. A secondary analysis looks at cases where appearance and geometry disagree strongly: $g_c$ high and $h_c$ low, or the reverse. These are relational counterparts of the shape-versus-texture discordant cases in Lesion-Centric Latent Phenotypes. We ask whether they concentrate reader disagreement or label errors. VinDr-CXR's three-reader training annotations make this checkable.

Baselines

Naive reference: the unchanged detector. The first reference is the Faster R-CNN (feature-pyramid ResNet-50) from ChestX-Det10, keeping its own class scores. The paper reports 40.2 overall AP50, with 26.1 for pneumothorax, 23.3 for nodules, 53.7 for consolidation and 66.2 for emphysema. We retrain it on our splits and first check that we reproduce roughly these numbers. Everything else re-scores the same proposals. Beating it means a higher overall AP50, with the improvement located in anatomy-defined classes.

Strongest end-to-end competitor. This baseline re-scores the same proposals with an unconstrained head. It sees the full-image feature map plus the box, has matched parameter count, and gets the same hyperparameter search budget as our model. This is the true threat, because it can in principle learn relative geometry implicitly. It stands in for the unconstrained networks that lost to structural bottlenecks in Pisov et al. (2.92 mm versus 1.54 mm error) and Yang et al. (AUC 0.7986 versus 0.9590). Those gains are in other tasks and do not transfer automatically. We count ourselves as beating it if we are within one AP50 point in-distribution and better under cross-hospital transfer, while passing the faithfulness test it fails.

Clinical rule baseline. This is a hand-coded measurement rule per class with a tuned threshold or a shallow tree. Examples are a cardiothoracic-ratio threshold for cardiomegaly and a costophrenic-angle overlap rule for effusion. It uses the same descriptors fed to logistic regression without the appearance channel.

Ablations isolating the key ingredient. Each ablation removes or replaces one part of the model, with every other setting held fixed:

  • replace the relative descriptors with absolute normalized box coordinates;
  • drop the anatomy masks and keep only pairwise geometry between candidate findings, which is the LinkNet-style setup;
  • keep the appearance channel alone;
  • keep the geometry channel alone;
  • a shuffled-anatomy control, where each image receives another patient's masks.

If the shuffled control loses little, the model is not actually using anatomy, and our claim fails.

No reported result was found on Lacuna for VinDr-CXR detection baselines, so in-hospital VinDr numbers will be established by our own runs of all methods.

Why not just…?

Why not just train an end-to-end detector and explain it with Grad-CAM? Heat maps in this setting are coarse. ChestX-Det10 notes they miss the extent and number of small lesions. The misalignment benchmark shows that deep-layer explanations shift when only outside pixels change (location change 24.0, activation change 23.7 for ProtoPNet). Accuracy is the real threat here: an end-to-end head may match us. The out-of-region perturbation experiment below is designed to separate the two approaches on faithfulness even if accuracy ties.

Why not just compute clinical rules like the cardiothoracic ratio directly from CheXmask? For cardiomegaly this is a genuine threat. A ratio threshold may be as good as anything learned, and we will say so if it is. Rules do not exist, or are not crisp, for most findings, and they ignore appearance entirely. Hand-crafted features trailed learned representations badly in breast ultrasound (radiomics AUC 0.774 versus 0.982 in Lesion-Centric Latent Phenotypes). The rule baseline is reported per class, so readers can see exactly where learning adds value.

Why not just give the detector absolute coordinates? A detector already sees position. Absolute coordinates, however, change with patient positioning, field of view and projection, while anatomy-relative quantities do not. In HO-RCNN, adding a relative interaction stream raised HICO-DET mAP from 5.73% to 7.30%. The absolute-coordinate ablation, evaluated specifically under cross-hospital transfer, settles the question here.

Why not just prompt a vision-language model with box coordinates? ChatRef shows that GPT-4 can reason over textual box geometry (75% on a RefCOCO sample). It also shows that weaker models collapse (23% for GPT-3.5), and that such reasoning inherits every detection error. The LLM prompting baseline in Lesion-Centric Latent Phenotypes reached only 59.1% on discordant cases. We include a small prompted baseline on a held-out subset: detector boxes plus our descriptors, serialized as text. This tests whether the learned $h_c$ adds anything beyond giving the numbers to a language model.

Proposed Experiments

Experiment 1: per-class detection re-scoring. We re-score proposals on ChestX-Det10 and report AP50 per class. The decisive check is the pattern, not the average. We predict significant gains for pneumothorax, effusion and atelectasis. We predict near-zero change for nodule, mass and calcification. A uniform gain, or gains concentrated in texture-defined classes, would mean the geometry channel is acting as generic extra capacity, which contradicts our claim.

Experiment 2: cross-hospital transfer. We train on VinDr-CXR for the overlapping finding classes and test on ChestX-Det10's NIH images without fine-tuning. VinDr-CXR supplies Vietnamese hospitals and a test set labeled by a five-radiologist consensus. The prediction is that the geometric model's AP50 drop is smaller than the end-to-end head's. We also stratify by anteroposterior versus posteroanterior projection, where CheXmask documents shift.

Experiment 3: faithfulness. We adapt the location, activation and rank change metrics of the misalignment benchmark. We apply patch perturbations and inject synthetic text tags strictly outside the crop and the anatomy masks. We then report the change in class score and in explanation rank for all methods. Our model should show exactly zero change except when a mask changes. We separately report how often the segmenter's masks move under these edits, and by how much. This is the honest bound on the guarantee.

Experiment 4: relation grounding. On the Chest ImaGenome gold subset, we test whether $h_c$'s descriptors alone predict the report-derived "located at" anatomy and laterality of findings. This checks that the geometric explanations agree with how radiologists describe location.

Experiment 5: discordance analysis. On VinDr-CXR training data we test whether the model's appearance–geometry discordance predicts disagreement among the three readers. This checks whether the decomposition is clinically meaningful and not just an architectural convenience.

The project fails in three identifiable ways:

  • The shuffled-anatomy control performs as well as real anatomy.
  • Gains do not follow the anatomy-defined versus texture-defined split.
  • The end-to-end head transfers as well as our model while also passing the perturbation test.