Framing image registration as a landmark detection problem for better representation of clinical relevance

Diana Waldmannstetter, Benedikt Wiestler, Julian Schwarting, Ivan Ezhov, Marie Metz, Spyridon Bakas, Bhakti Baheti, Satrajit Chakrabarty, Jan S. Kirschke, Rolf A. Heckemann, Marie Piraud, Florian Kofler, Bjoern H. Menze

Abstract

Traditional evaluation of medical image registration relies on Target Registration Error (TRE), which often fails to account for human annotation noise and clinical significance. This paper proposes the Landmark Hit Rate (HitR), a binary metric that treats registration as a detection task. By defining a confidence zone based on inter-rater variance, HitR provides a more robust and clinically relevant measure of algorithm performance.

Introduction

Image registration is a cornerstone of medical imaging, essential for aligning multi-temporal or multi-modal scans. Historically, the field has relied on the Target Registration Error (TRE)—the Euclidean distance between predicted and ground-truth landmarks—as the gold standard for evaluation. However, TRE has significant drawbacks. First, it ignores "label noise"; human experts rarely agree on exact landmark coordinates, making sub-millimeter precision requirements arbitrary. Second, it lacks clinical context; a large TRE in a non-critical area might be less problematic than a smaller error in a high-stakes region.

Methods: The Landmark Hit Rate (HitR)

To address these issues, we propose framing registration as a landmark detection problem. Instead of a continuous distance metric, we introduce the Landmark Hit Rate (HitR).

Defining the "Hit"

A registration is classified as a "hit" if the predicted landmark falls within a predefined radius ($r$) of the reference landmark, forming a Region of Interest (ROI) or "confidence zone." If the prediction falls outside this zone, it is a "miss."

Visual representation of a 'Hit' versus a 'Miss' in the HitR framework. A hit occurs when the predicted landmark (pink) falls within the confidence zone (red circle) centered on the reference (yellow).

Accounting for Inter-rater Variance

To ensure the metric reflects human uncertainty, the ROI radius is ideally derived from inter-rater variance—the "circle of disagreement" among multiple experts. For datasets with limited annotations, we employ a "silver standard" method using Median Absolute Deviation (MAD) to extrapolate robust radii. The final HitR score is the ratio of hits to total landmarks.

Results and Insights

The metric was validated using the BraTS-Reg dataset, comprising 50 paired brain MRI quartets with 739 anatomical landmarks.

The "Crossing Curves" Phenomenon

By plotting HitR against varying radii, we observed "accuracy curves." Some algorithms demonstrated "brittleness," performing well at high precision but failing to improve as tolerance increased. Conversely, "robust" algorithms showed significant gains as the ROI expanded, suggesting better clinical reliability.

TRE vs. HitR

Our analysis revealed only a moderate correlation between TRE and HitR (Pearson $r = -0.49$). This discrepancy highlights that algorithms topping TRE-based leaderboards may not necessarily be the most clinically dependable tools, as they might be optimized for sub-resolution precision at the expense of overall robustness.

Discussion and Future Work

While HitR offers a more clinical perspective, it has limitations. Currently, ROIs are modeled as simple spheres, whereas anatomical uncertainty is often directional. Future iterations could incorporate non-spherical topologies. Additionally, the binary nature of HitR makes it non-differentiable, preventing its direct use as a loss function in deep learning. Developing "signed distance" variants could bridge this gap.

In conclusion, HitR shifts the focus from raw distance to clinical utility, providing a framework that better aligns algorithmic evaluation with the needs of medical practitioners.

Summary updated

Abstract

Nowadays, registration methods are typically evaluated based on sub-resolution tracking error differences. In an effort to reinfuse this evaluation process with clinical relevance, we propose to reframe image registration as a landmark detection problem. Ideally, landmark-specific detection thresholds are derived from an inter-rater analysis. To approximate this costly process, we propose to compute hit rate curves based on the distribution of errors of a sub-sample inter-rater analysis. Therefore, we suggest deriving thresholds from the error distribution using the formula: median + delta * median absolute deviation. The method promises differentiation of previously indistinguishable registration algorithms and further enables assessing the clinical significance in algorithm development.
BibTeX
@misc{waldmannstetter2023framing,
  author = {Diana Waldmannstetter and Benedikt Wiestler and Julian Schwarting and Ivan Ezhov and Marie Metz and Spyridon Bakas and Bhakti Baheti and Satrajit Chakrabarty and Jan S. Kirschke and Rolf A. Heckemann and Marie Piraud and Florian Kofler and Bjoern H. Menze},
  title = {{Framing image registration as a landmark detection problem for better representation of clinical relevance}},
  year = {2023},
  eprint = {2308.01318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.01318},
}
Figure