Anatomy Likelihood Channels for Native-Resolution Landmark Detection

Medical landmark detection finds a fixed set of anatomical points, such as skull or knee reference points on an X-ray, to sub-millimetre accuracy. It is usually done by predicting a "heatmap", a coarse grid of scores whose peak marks the point. Two separate losses of precision follow from this design. The first is quantization error: the answer is snapped to a grid cell that may be four or more original pixels wide. The second is resolution degradation: the network shrinks the image as it processes it, so faint, small structures are blurred away. The obvious fix is to work at the scanner's full resolution. It does not work cleanly. The strongest direct-regression method reports that accuracy improves up to 1024×1024 input and then stalls or gets worse at 1280×1280 (Du-CeLR).

We propose that this plateau is mainly a problem of ambiguity, not of pixels. At full resolution, a patch of fixed size contains more local structures that look like the target: bone edges, overlapping contours, osteophytes. The network then more often fires on the wrong one. Sub-pixel decoding cannot fix a peak placed on the wrong structure. Our method runs a cheap coarse model first. From its output and a statistical shape model, we build an explicit anatomical likelihood map for each landmark: a map of where that point should be, given where all the other points are. A second-stage network works on a native-resolution crop and receives this map as an extra input channel next to the image pixels. The two-sentence version: high-resolution landmark errors are dominated by picking the wrong peak, so tell the fine-scale network where anatomy says the point should be before it looks at the pixels. Done this way, native resolution should keep paying off instead of plateauing.

Core Claim

We make two linked claims. The first is diagnostic. If landmark error is split into three parts, the parts behave differently as resolution rises:

  • Grid residual: the error an oracle heatmap would still have at the chosen stride.
  • Local imprecision: error within the correct peak.
  • Mode error: choosing a wrong peak.

We expect the grid residual to shrink as resolution rises, while the mode-error share grows, and we expect mode error to explain the Du-CeLR plateau.

The second claim concerns the method. Conditional anatomical likelihood maps, supplied as an input channel to a native-resolution crop network, remove most of that mode error. They do this without the brittleness that hand-built priors usually bring.

The grid residual has a simple floor, which we use as a reference line. If the true point falls uniformly inside a heatmap cell of side $s$ (in mm), the root-mean-square radial snapping error is

$$ \epsilon_q = s/\sqrt{6}, $$

because each axis contributes the variance of a uniform variable, $s^2/12$. We compute $s$ from each dataset's pixel spacing and each model's stride. This tells us how much of the reported mean radial error (MRE, the average distance between predicted and true points) quantization could possibly account for.

Relation to Prior Work

The mechanism for injecting domain knowledge comes from work that feeds expert-derived maps as extra input channels. Frank et al. supplied pleural-line and artifact masks alongside lung ultrasound and needed less training data. The ECG rhythm work turned detected R-, P- and T-wave positions into likelihood maps and concatenated them to the raw leads. That same paper is also a warning. Its hand-built pipeline was costly enough to time out on the hidden test set (score −0.406 versus 0.630 in cross-validation), and hand-crafted filtering hurt compared with training on raw signals. We take two lessons from it. The prior must be cheap: here it is a single linear-Gaussian conditional per landmark. And the network must be trained not to trust the prior blindly: we randomly inflate and shift it during training.

The quantization literature supplies our decoding tools. It also tells us which ones will not work alone:

This constrains our design. We decode with a soft-argmax restricted to a window around the chosen peak. We treat the real problem as choosing the right peak, which no decoding rule solves by itself.

On the resolution side, three lines of work each cover part of what we want:

  • The HRNet-W48 + Pixel Shuffle + DARK pipeline keeps features at high resolution and refines peaks with a Taylor expansion. Its authors state that it fails on obscured jaw edges because it knows no anatomical rules.
  • GU2Net multiplies a local heatmap by a dilated-convolution global heatmap. This is a learned, implicit prior applied at the output with plain argmax. It is weak on low-contrast soft tissue.
  • Du-CeLR, the closest existing work, spans all three directions. A transformer "reference encoder" learns relations between landmarks. It then refines by sampling backbone features near coarse coordinates. Its relations are implicit, it samples downsampled backbone maps rather than native pixels, and it hits the resolution plateau.

We differ from Du-CeLR in three ways: the prior is explicit and leave-one-out conditional, it enters as an input to the fine network rather than as attention, and the fine stage sees native pixels. We also offer a testable explanation of the plateau Du-CeLR reports. AFPF regression-voting handles quantization with offset voting but encodes no global layout.

Research Program

Stage one. A small whole-image heatmap network runs at 512 pixels and gives rough positions for all $K$ landmarks.

The prior. From training annotations we fit a Gaussian point-distribution model of the joint landmark layout, after aligning shapes for translation, rotation and scale. For landmark $k$, we condition this Gaussian on the stage-one positions of all other landmarks. The result is a mean $\mu_k$ and covariance $\Sigma_k$ that never use landmark $k$'s own possibly-wrong estimate. Conditioning is a closed-form linear operation. We also down-weight other landmarks whose stage-one confidence is low, so one bad point cannot poison the rest.

Stage two. We crop a fixed-size window at native pixel spacing around $\mu_k$. We render $\mathcal{N}(\mu_k, \Sigma_k)$ as a likelihood channel on that crop. The fine network takes pixels plus likelihood channel and predicts a heatmap at stride 1 or 2 using a Pixel Shuffle head, following the SR-head design. We decode with windowed soft-argmax.

Training. We use the true stage-one error distribution plus random covariance inflation and mean jitter. This teaches the network to overrule the prior when the image evidence is clear, which matters most on deformed anatomy.

Error decomposition. We label a prediction as a mode error when its error exceeds a radius $r$ and the heatmap has a secondary local peak within $r$ of the truth. We report $r$ at 2 mm and 3 mm. We apply this analysis to our models and to reimplemented baselines across input resolutions from 512 up to native.

Baselines

Do-nothing reference. A standard heatmap network at 1024 input and stride 4, decoded with plain argmax. This gives the floor and lets us check measured grid residual against $\epsilon_q$.

Strongest high-resolution heatmap system. The HRNet-W48 + SR head + DARK pipeline. It reports MRE 1.5176 mm and 75.7% SDR@2mm on CLDetection2023, where SDR is the fraction of points within the stated distance. It is the strongest competitor because it attacks both resolution loss and quantization directly and uses no anatomy. If our gain over it holds, knowledge adds something beyond pixels. We run it in MMPose with our splits, augmentation and GPU-hour budget, tuning each method over equal hyperparameter grids.

Strongest method overall. Du-CeLR, which reports MRE 1.01 mm and 89.51% SDR@2mm on ISBI 2015 Test 1, and MRE 0.9372 mm on ISBI 2023. It is the most direct competitor on the question of whether explicit anatomy beats learned relations. If code is unavailable, we reimplement it and report both our numbers and theirs, clearly separated.

Secondary references. - AFPF voting: ISBI 2015 Test 1 MRE 1.17 mm / 86.67% SDR@2mm; Test 2 MRE 1.48 mm / 75.05%. - KNEEL: OAI PCK@2mm 85.36%, outlier rate 0.53%. - GU2Net: head MRE 1.54 mm; an implicit output-level prior.

Ablations isolating the key ingredient. All ablations keep the same two-stage cascade and native-resolution crops.

  • Crop without prior: the likelihood channel is removed. This is the most important ablation.
  • Prior as post-hoc reweighting: the heatmap is multiplied by the prior after prediction, GU2Net-style.
  • Prior without robustness training: no covariance inflation or jitter.
  • Learned relations: a Du-CeLR-style attention module produces the crop centre instead of the shape model.

What counts as beating a baseline. We require four things together:

  • MRE reduction beyond the paired-bootstrap confidence interval against the strongest reimplemented baseline.
  • A lower mode-error rate.
  • No loss on KL-4 knees.
  • For the central claim specifically, error that keeps falling beyond 1024 pixels while the crop-without-prior ablation stalls.

Why not just…?

Why not just use local (windowed) soft-argmax, or DARK, to fix multimodal heatmaps? These decoders fix what happens inside a chosen peak. They pick the peak by maximum score, so they inherit every wrong-peak choice. Heatmap regression without soft-argmax identifies averaging between peaks as a failure, not wrong-peak selection. We use windowed soft-argmax in every arm, so the decomposition shows directly how much error remains as mode error after the decoding fix.

Why not just crop native-resolution patches around a coarse prediction, without any prior? This is the real threat. A coarse-to-fine crop alone may capture most of the resolution gain. If the crop-without-prior ablation matches the full model at all resolutions, our anatomical contribution is void and we will report it that way. Our hypothesis predicts a specific divergence instead: the two arms agree at low resolution, the prior-free arm's mode error grows at native resolution, and the full model's does not.

Why not just fit a shape model after prediction to reject implausible points? Post-hoc correction can only move a point toward the shape-model mean, or choose among peaks the network already produced. When the correct structure is faint, as in soft tissue or obscured jaw edges, the network may never produce that peak. An input channel can change what the network attends to. The post-hoc-reweighting ablation tests this directly, and we expect the gap to be largest on low-contrast landmarks.

Why not just let a transformer learn inter-landmark relations, as Du-CeLR does? Learned relations need data to fit a layout that a closed-form Gaussian gives for free from 150–600 annotated images. Learned relations also cannot easily be widened on demand for unusual patients. The learned-relations ablation compares them at matched cost. The prior-corruption test below checks whether the explicit prior is easier to override when it is wrong.

Proposed Experiments

Datasets. We train and evaluate on four datasets:

  • ISBI 2015: 19 landmarks, Test 1 and Test 2 reported separately.
  • MICCAI CLDetection2023: 38 landmarks, a harder and more diverse set.
  • OAI knees, stratified by Kellgren-Lawrence grade 0–4, a scale of osteoarthritis severity. This is the deformity stress test.
  • The GU2Net hand X-ray set, for a different anatomy.

Metrics are MRE, SDR at 2, 2.5, 3 and 4 mm, PCK and >10 mm outlier rate on OAI, measured grid residual, and mode-error rate.

Check 1: the resolution sweep. Every arm is trained at 512, 768, 1024, 1280 and native resolution. Our central claim holds only if the crop-without-prior arm reproduces the Du-CeLR-style plateau, that plateau lines up with rising mode error, and the full model removes both.

Check 2: deformity. On OAI by KL grade, the prior must not hurt at KL 4. If it does, the robustness training has failed, and we will say the method is limited to near-normal anatomy.

Check 3: prior corruption. At test time we shift $\mu_k$ by 1–5 mm. We measure how far predictions follow the shift on clear bony landmarks versus faint ones. We want small following on clear landmarks and larger on faint landmarks, which would show the network weighs prior against evidence rather than copying the prior.

Check 4: cross-device transfer. We train on ISBI 2015 and test on CLDetection2023 without retraining. Anatomy changes less across scanners than texture does, so the prior's gain should be larger here than within one dataset.

Cost. We report wall-clock inference per image against the HRNet-W48 pipeline. The ECG precedent shows that a knowledge pipeline too slow to deploy is not a success.