Leave-One-Out Landmark Consensus for Occluded 3D Anatomy

Finding anatomical landmarks in a CT scan, such as the center of each vertebra or a point on the inner ear, becomes hard exactly where it matters most clinically. Metal screws, fractures and surgical voids erase the local image evidence, so the model has to infer the hidden point from the rest of the anatomy. The best 2D method for this, used on faces, compares every image patch against a "consensus" version of that patch assembled from all the other patches. That comparison costs the square of the number of patches. In a 3D volume the patch count grows with the cube of the resolution, so the cost grows with its sixth power, and the idea cannot be ported directly. The usual workarounds, coarse patches or dropping tokens, blur the millimeter detail that landmark localization needs.

We propose to run the consensus at the level of landmarks rather than patches. A small set of $K$ landmark queries (about 26 for a spine) each read the whole coarse volume. Each landmark's position is then re-predicted from all the other landmarks, both their locations and their image features. When a landmark's own estimate disagrees with this leave-one-out consensus, the model treats it as occluded and leans on the consensus. The corrected position then decides where to spend full-resolution computation. Cost falls from quadratic in voxel tokens to linear in tokens times $K$. Occlusion detection becomes a cheap $K \times K$ operation. Resolution is spent only in $K$ fixed-size crops centered on positions that the consensus has already checked.

Core Claim

In 3D landmark localization, leave-one-out consensus among landmark queries can do three jobs that prior work handles separately. It detects occlusion, it recovers the occluded coordinates, and it steers where high-resolution refinement happens. We claim this beats three simpler options on scans with metal hardware or missing anatomy while keeping cost linear in volume size: detection followed by post-hoc shape fitting, occlusion augmentation alone, and patch-level consensus restricted to local windows. The claim is falsifiable. If a robust statistical shape model fitted to ordinary heatmaps matches it, the learned consensus adds nothing, and we will say so.

Position Relative to Prior Work

The mechanism comes from ORFormer. ORFormer gives each 2D patch a "messenger" token that gathers context from every other patch. It flags occlusion where patch and messenger disagree and repairs features using a heatmap codebook learned from unoccluded faces. It reports a normalized mean error of 3.86 on WFLW and 4.46 on COFW. We keep the disagreement test but move it from $N$ patches to $K$ landmarks. That change is what makes it affordable in 3D. It also changes what is being compared: coordinates and landmark-level features rather than raw appearance.

Two existing designs motivate the proposal by showing where current systems break.

  • Atlas Location Autocontext fits an affine atlas to coarse heatmaps and feeds it into a finer pass. It reaches 2.29 mm mean error on head CT against 2.20 mm between human annotators. Its global fit, however, is computed over all coarse predictions, so corrupted ones pull it off. Our consensus excludes each landmark from its own prediction and learns how much to trust each contributor.
  • SCE-UNETR's semantic-edge 2D/3D registration resists overlapping tissue with a hand-set inter-vertebral spacing loss and a separate optimizer. We learn the spacing relations end to end, in 3D, inside the network.

Three efficiency methods constrain the design.

  • Token Sparsification prunes 3D tokens and re-inserts early tokens for decoding. It keeps Dice but worsens the 95th-percentile Hausdorff distance, which tells us that pruning by "informativeness" damages boundaries. It also has no notion of a token being corrupted rather than redundant.
  • UNETR shows that large $16^3$ patches make global attention fit in memory at the price of fine detail.
  • D-Former alternates local-window and dilated attention. It reaches 88.83% Dice on Synapse with 44M parameters.

We use a D-Former-style backbone for the coarse stage. We do not rely on it for fine localization.

S4ND and LIP show that small targets need resolution to be kept, not recovered after downsampling. This is why our fine stage reads raw-resolution crops rather than upsampled coarse features.

Mechanism

The coarse encoder produces tokens at roughly 4 mm spacing. $K$ learned landmark queries cross-attend to these tokens and each outputs:

  • a local position estimate $p_k$,
  • an uncertainty $\sigma_k$,
  • a feature vector $f_k$.

A small transformer over the other $K-1$ landmarks, with landmark $k$ masked out, produces a consensus position

$$\hat{p}_k = g\big(\{(p_j, f_j, \sigma_j)\}_{j \neq k}\big).$$

A gate $\alpha_k = \mathrm{sigmoid}\big(h(\lVert p_k - \hat{p}_k \rVert / \sigma_k,\, f_k)\big)$ weighs the two into the fused estimate $\tilde{p}_k = (1-\alpha_k)\,p_k + \alpha_k\,\hat{p}_k$. Here $\alpha_k$ is the estimated probability that landmark $k$ is occluded, and $h$ is a small network.

A fine head then extracts a fixed $c^3$ crop at native resolution around each $\tilde{p}_k$ and regresses a residual offset. Total cost is linear in coarse tokens $N$ for the cross-attention, plus $O(K^2)$ for the consensus, plus $K c^3$ for refinement. None of these terms grows with the square of the volume.

Training uses synthetic occlusion. We paste simulated metal-streak patterns and intensity voids over randomly chosen landmarks, and the known mask gives a supervision signal for $\alpha_k$. The consensus head is trained to predict masked landmarks from the others. The fine head is trained on crops centered at $\tilde{p}_k$ plus jitter, so it tolerates consensus error.

One known failure mode of consensus methods is that they hallucinate an average shape when a landmark is deformed rather than hidden. Fractured vertebrae are the key stress test for this. The gate must learn to trust visible-but-displaced evidence, and we evaluate that case separately.

Research Program

We will build the model on spine CT and test on head CT. The first question is whether landmark-level consensus detects occlusion. We measure the area under the ROC curve of $\alpha_k$ against synthetic occlusion masks, and against radiologist-marked metal regions on real scans. The second question is whether it recovers coordinates better than post-hoc alternatives under matched compute. The third is whether consensus-guided crop placement is what preserves millimeter accuracy, as opposed to simply having a fine stage.

As a sanity check that we did not lose anything by moving from patches to landmarks, we will run a 2D version on WFLW and COFW. If it lands far above ORFormer's reported errors, the landmark-level design discards information that patch-level consensus uses.

Baselines

Naive reference. The do-nothing reference is a D-Former-style heatmap regressor at the coarse resolution, followed by the same fixed-size high-resolution crop refinement, with no consensus. It is trained with identical synthetic-occlusion augmentation. This isolates everything except the consensus and gate. Beating it means lower localization error and higher identification rate on the metal-implant and synthetic-occlusion subsets of VerSe, with no loss on clean scans.

Strongest existing on VerSe. The strongest reported system is the VerSe 2020 winning model in the VerSe benchmark. It reports a 96.6% identification rate and 91.72% Dice; the 2019 top performer reports 94.25% identification. We will evaluate on the public VerSe 2020 split and compare identification rate against the reported 96.6%. We expect the margin to show mainly on scans with implants or transitional vertebrae, which we will report as separate strata. We cannot retune that system, so this comparison is against published numbers only. All other comparisons are retrained by us.

Strongest existing on head CT. This is the two-pass atlas autocontext network (Atlas Location Autocontext), at 2.29 mm mean radial error on its 201-scan benchmark. If that dataset is accessible, we will reimplement the network, train it with the same augmentation and GPU budget, and compare on clean scans and on scans with inserted occluders. Beating it means staying near 2.29 mm on clean data while degrading less under occlusion. The atlas fit's known weakness predicts a steeper degradation for it.

Ablations. The ablations that isolate the key ingredient are:

  • a gate fixed to zero, meaning local evidence only;
  • a consensus using coordinates only without image features, which amounts to a learned shape model;
  • crops centered on $p_k$ instead of $\tilde{p}_k$.

Fairness conditions. Every trained model uses the same data splits, augmentation, a fixed search budget of 20 hyperparameter trials, and the same single-GPU memory cap.

Why not just…?

Why not just detect landmarks with a standard network and fit a robust statistical shape model afterwards? This is the most serious threat. Fitting a PCA shape model with outlier rejection over detected points is cheap and well understood, and it may handle isolated missing vertebrae well. Its weaknesses are twofold. It sees coordinates but not image features, so it cannot tell a displaced fracture from an occlusion. It is also fitted after the fact, so it cannot redirect where the fine stage looks. We will run exactly this baseline on the naive model's heatmaps, together with the coordinates-only consensus ablation. If either matches the full model on the implant and fracture strata, the proposal's central claim fails, and we will report the shape-model result as the recommended method.

Why not just train an ordinary 3D network with heavy occlusion augmentation? Augmentation teaches a network to ignore corrupted voxels. When a whole vertebra is erased, though, recovery requires reasoning over distant vertebrae. The one-label-shift errors documented in VerSe suggest local networks lack this reasoning. Our naive baseline receives identical augmentation, so any gap between it and the full model measures what consensus adds beyond augmentation.

Why not just port ORFormer's patch messengers into D-Former's local windows? This keeps cost linear but limits consensus to a window. Recovering an occluded vertebra needs evidence from vertebrae several windows away. We will implement this variant and compare it on synthetic occlusions of increasing extent, from one to three adjacent vertebrae. If windowed patch consensus holds up as well as landmark consensus at large extents, the landmark-level restriction is unnecessary.

Why not just use a coarse-to-fine cascade? A plain cascade is our naive baseline. Its known failure is that a coarse miss under occlusion sends the fine stage to the wrong place, as observed for Atlas Location Autocontext. The crop-placement ablation, centering on $p_k$ versus $\tilde{p}_k$, tests exactly this failure.

Proposed Experiments and Decisive Checks

Metrics and settings. On VerSe 2019 and 2020 we report:

  • vertebra identification rate;
  • centroid localization error in millimeters, split into clean scans, scans with metal fixation, scans with fractures, and scans with transitional vertebrae;
  • peak GPU memory and runtime per volume.

We add controlled synthetic occlusion by erasing or streaking one to three vertebrae on held-out clean scans, where ground truth is exact. On the head CT benchmark we report mean radial error with and without inserted occluders.

Decisive checks.

  • Occlusion recovery. On the synthetic and metal strata, the full model must beat both the robust shape-model baseline and windowed patch consensus in localization error, at equal or lower memory.
  • Gate discrimination. The gate's occlusion AUROC should be high on synthetic masks. On fractured-but-visible vertebrae, $\alpha_k$ should stay low. If it does not, the model is averaging away real deformation, which would confirm the hallucination concern.
  • Compute scaling. Memory should grow roughly linearly with volume size as we increase field of view, while a full-attention coarse encoder grows much faster.
  • 2D sanity check. The 2D variant should come within a small margin of ORFormer's reported 3.86 on WFLW and 4.46 on COFW. If it does not, the landmark-level design discards information and must be revised before any 3D claim stands.