Report-Anchored Angle Costs for Chest X-ray Phrase Grounding

When a radiologist writes "small left apical pneumothorax" or "blunting of the right costophrenic angle," the sentence fixes some directions of the finding's location much more tightly than others. "Left" settles which side of the chest. "Apical" places it near the top but says little about how far from the midline. Models that turn such a phrase into a rectangle on the X-ray ignore this structure. During training they shrink the gap between the predicted and true box centers equally in every direction. This proposal asks whether the box-fitting loss should instead correct the direction the report pins down first, and only then the direction it leaves loose. The direction would be read off the anatomical words a clinical text extractor already pulls from the report.

The idea modifies an existing loss. Scylla-IoU (SIoU), a box-fitting loss from object detection, adds an "angle cost." This cost measures the angle between the line joining the two box centers and the nearest image axis, and it pushes the prediction to remove the smaller axis offset first. We replace that fixed image axis with a reference direction computed from the report's anatomy entities, of the kind that RadGraph-style extraction produces. When the reference direction is the image x-axis, the new loss reduces exactly to SIoU, so the comparison against SIoU is a clean ablation. The question we test is narrow. Does a text-derived axis help grounding accuracy and training stability on small medical grounding datasets, compared with a fixed image axis, a generic distance loss, or simply giving the model the location words as input?

Core Claim

In phrase grounding, a model reads a phrase and outputs the box it describes. On chest X-rays the uncertainty of a finding's position, given its description, is strongly anisotropic. Anisotropic here means the box center is far more predictable along one direction than along the perpendicular one. That direction depends on the anatomical term, not on the image axes.

The principal axes of this uncertainty can be estimated from training boxes grouped by anatomy term. They can then be used as the reference frame of an SIoU-style angle cost. Two things follow from this design. The loss orders its corrections along a frame the text actually determines. The optimizer also loses a degree of freedom exactly where the text supplies one. We expect this to matter most for elongated, edge-hugging findings such as pneumothorax and pleural effusion. We expect it to matter least for diffuse or central findings such as cardiomegaly. We also expect it to matter more on rotated portable films than on upright posteroanterior films.

Prior Work and Where This Sits

The angle-aware regression literature supplies the method. SIoU combines angle, distance, shape and overlap costs. Its angle cost acts on the center-offset vector relative to the nearest image axis. It was adopted inside larger YOLOv5 redesigns, namely MSS-YOLOv5 on PASCAL VOC (He et al.) and an aircraft detector on GF-3 synthetic aperture radar images (Wang et al.). These papers motivate the proposal, but they also constrain how much it can claim.

Their reported gains, +2.4/+2.9 mAP on VOC and mAP50 rising from 0.873 to 0.931 on GF-3, come from models that changed several modules at once. Neither isolates the effect of the loss. We therefore treat "SIoU helps" as a hypothesis to retest on medical data, not as established. Related work separates angular error into its own loss term (Niu et al.) or uses explicit vector angles when image gradients are weak, as in thermal infrared imaging (Wang). Both still measure angles against fixed image coordinates. That is the "axis-alignment bias" our method removes.

Medical grounding supplies the application and the strongest systems. MedRPG (paper) grounds a pre-extracted phrase. It uses a ResNet-50 image encoder and a BERT text encoder, joined by a 3-layer transformer that fuses the two. A dedicated regression token outputs the box. Its contrastive alignment terms discourage location shortcuts. uMedGround (paper) grounds phrases straight from a full report. A multimodal large language model emits a special box token that feeds a decoder built on the Segment Anything Model, trained with Smooth L1 plus Generalized IoU (GIoU) losses and several candidate boxes per finding.

Both systems use the image frame for every finding. We keep their architectures and change only how the box error is decomposed. ELVIS (paper) preserves within-image and within-report similarity structure during vision-language pretraining. It reports localization as heatmap contrast, not as boxes. We treat it as a possible backbone, not as a competitor on box regression.

Clinical extraction supplies the input to the frame. RadGraph (paper) labels Anatomy and Observation entities, links them with Located At and Modify relations, and reaches 0.94 entity F1 on MIMIC-CXR. That makes it a reliable source of the anatomy term attached to each finding. Its reported drop when moving from MIMIC-CXR to CheXpert, and its relation agreement of only κ = 0.397, are real constraints. We plan for extraction errors and measure their effect instead of assuming clean anatomy terms.

Research Program

The first step builds the reference frames. We run the RadGraph model on each MS-CXR phrase, or on the source report sentence, and keep the Anatomy entity linked to the finding by Located At, along with its Modify children such as "left," "apical," or "lateral."

For each normalized anatomy term $a$ with enough training examples, we take the normalized ground-truth box centers. We compute their covariance and its eigenvectors. The leading eigenvector $u_a$ is the loose direction and the second, $u_a^\perp$, is the tight direction. To make the frame follow the patient rather than the image, we optionally rotate it by the tilt of the line joining the two lung apices. That tilt comes from an off-the-shelf lung segmentation. Rare or missing terms fall back to the observation-level frame and then to the image axes.

The second step defines the loss. Let $d$ be the vector from the predicted box center to the true box center. Let $\alpha_a \in [0, \pi/4]$ be the angle between $d$ and whichever of $u_a$ or $u_a^\perp$ is closer. The angle cost keeps SIoU's form,

$$ \Lambda_a = 1 - 2\sin^2\!\left(\alpha_a - \tfrac{\pi}{4}\right), $$

which is zero when the offset lies along one frame axis and largest along the diagonal. The distance and shape costs are computed in the rotated frame. Each is weighted by the inverse of the per-axis spread $\sigma_a^\perp, \sigma_a^\parallel$ from the covariance, so errors along the tight axis cost more. Setting $u_a$ to the image x-axis with equal weights recovers SIoU exactly. The frame depends only on the text and the training set, never on the target box at test time, so the model cannot cheat by moving the frame.

The third step plugs the loss into MedRPG. MedRPG is small, has public-benchmark numbers, and isolates the loss more cleanly than an end-to-end report model. We run it second inside a uMedGround-style report grounding head. That shows whether the effect survives when phrases are themselves predicted. Throughout, we log center error separately along the tight and loose axes. This records whether the loss changes where errors go, not just how large they are.

Baselines

The naive reference point uses the text alone. For each anatomy and observation pair, it predicts the mean training box. This shows how much localization the text determines without looking at the image. If our method's gains appear only on findings where this prior is already strong, the loss is mainly enforcing a location prior, and we will report it that way.

The strongest existing phrase-grounding system is MedRPG with its published loss. The file reports 69.86% accuracy at IoU > 0.5 on MS-CXR (MedRPG). We retrain it with the authors' settings to get our own reference and require our number to be within seed noise of 69.86% before comparing anything. For end-to-end report grounding, the file reports that uMedGround beats its baselines on AP10/30/50 and mIoU on MRG-MS-CXR and MRG-ChestX-ray8. No absolute value appears there, so no reported result was found on Lacuna for uMedGround's absolute MRG-MS-CXR accuracy. We will retrain it, or a faithful reimplementation of its box head, and report our own number.

The ablations that isolate the key ingredient change only the loss and keep everything else in MedRPG fixed. These are MedRPG with GIoU, with Distance IoU, with Cartesian SIoU, and with our report-anchored cost. A control assigns each phrase the frame of a randomly chosen other anatomy term. If shuffled frames match real ones, the benefit comes from anisotropic weighting, not from the text.

Every variant gets the same splits and the same backbone initialization. Each gets an identical tuning budget for its own loss weights: a fixed grid of at most 12 configurations chosen on validation. Each runs with five seeds and trains to the same epoch cap. Beating Cartesian SIoU means a gain in mean accuracy at IoU 0.5 whose 95% bootstrap interval excludes zero, or a statistically clear reduction in tight-axis error with no loss in accuracy.

Why not just…?

Why not just swap GIoU for Cartesian SIoU in MedRPG? This is the main threat. Most of chest X-ray anatomy runs roughly parallel to the image edges on upright films, so the anatomical frame and the image frame often nearly coincide. The ablation answers this directly. We break results down by finding type and by film rotation, measured with the apex-line tilt. If report-anchored frames beat Cartesian SIoU only on rotated portable films, or only on edge-hugging findings, that is a narrower result, and we will state it in those terms. If they never beat it, the contribution reduces to the first controlled test of whether SIoU helps medical grounding at all. The confounded natural-image results cannot settle that question.

Why not just feed the extracted anatomy words to the model as extra input? A model given "left apical" as text might learn the same anisotropy without any change to the loss. We test this by appending the RadGraph anatomy span to MedRPG's input and training it with GIoU. If this matches our method, the loss is redundant and the simpler input change is the better recommendation. Our expectation is that the input conveys where the box should be, while the loss shapes how errors are corrected during training. That matters with only about 1,153 MS-CXR examples, but it is an empirical question.

Why not just let the model learn per-axis uncertainty itself? A head that predicts a separate variance for each axis from the fused image-text features would learn anisotropy without any clinical extraction. We include this variant, trained with a Gaussian negative log-likelihood on the center plus GIoU. It is the cleanest test of whether the RadGraph-derived frame adds anything beyond learned anisotropy. A tie would favor the simpler model.

Why not just train GIoU longer? If SIoU's benefit is mostly faster convergence, as the detection papers describe it, longer GIoU training should close the gap. We therefore report accuracy at matched epochs and at convergence, and count only the converged numbers as wins.

Proposed Experiments

The primary benchmark is MS-CXR, scored by accuracy at IoU 0.5 and by mIoU, with five seeds and bootstrap intervals. ChestX-ray8's 984 hand-labeled boxes serve as a second in-domain test. VinDr-CXR provides a cross-institution stress test (VinDr-CXR). It has radiologist consensus boxes on its test set but no reports, so phrases are built from class names plus anatomy templates. We flag this as easier text than real reports.

Four checks decide the outcome. The first asks whether report-anchored frames beat Cartesian SIoU on pneumothorax and pleural effusion while tying on cardiomegaly, which is the pattern anisotropy predicts. The second applies synthetic rotations of ±15°. If gains appear there, the patient-aligned frame should keep its accuracy while Cartesian SIoU degrades. The third compares real frames with shuffled frames. The fourth corrupts a growing fraction of the RadGraph anatomy terms. Since RadGraph's agreement scores show extraction is imperfect, accuracy should fall back smoothly toward the Cartesian SIoU baseline, not collapse below it.

A positive result on all four would show that clinical text extraction can do more than produce labels: it can set the geometry in which visual localization is optimized. A negative result on the first check, with ties throughout, would be a useful finding too. It would show that angle-aware box losses do not carry over to small medical grounding datasets.