Abstract
Traditional evaluation of medical image registration relies on Target Registration Error (TRE), which often fails to account for human annotation noise and clinical significance. This paper proposes the Landmark Hit Rate (HitR), a binary metric that treats registration as a detection task. By defining a confidence zone based on inter-rater variance, HitR provides a more robust and clinically relevant measure of algorithm performance.
Introduction
Image registration is a cornerstone of medical imaging, essential for aligning multi-temporal or multi-modal scans. Historically, the field has relied on the Target Registration Error (TRE)—the Euclidean distance between predicted and ground-truth landmarks—as the gold standard for evaluation. However, TRE has significant drawbacks. First, it ignores "label noise"; human experts rarely agree on exact landmark coordinates, making sub-millimeter precision requirements arbitrary. Second, it lacks clinical context; a large TRE in a non-critical area might be less problematic than a smaller error in a high-stakes region.
Methods: The Landmark Hit Rate (HitR)
To address these issues, we propose framing registration as a landmark detection problem. Instead of a continuous distance metric, we introduce the Landmark Hit Rate (HitR).
Defining the "Hit"
A registration is classified as a "hit" if the predicted landmark falls within a predefined radius ($r$) of the reference landmark, forming a Region of Interest (ROI) or "confidence zone." If the prediction falls outside this zone, it is a "miss."
Accounting for Inter-rater Variance
To ensure the metric reflects human uncertainty, the ROI radius is ideally derived from inter-rater variance—the "circle of disagreement" among multiple experts. For datasets with limited annotations, we employ a "silver standard" method using Median Absolute Deviation (MAD) to extrapolate robust radii. The final HitR score is the ratio of hits to total landmarks.
Results and Insights
The metric was validated using the BraTS-Reg dataset, comprising 50 paired brain MRI quartets with 739 anatomical landmarks.
The "Crossing Curves" Phenomenon
By plotting HitR against varying radii, we observed "accuracy curves." Some algorithms demonstrated "brittleness," performing well at high precision but failing to improve as tolerance increased. Conversely, "robust" algorithms showed significant gains as the ROI expanded, suggesting better clinical reliability.
TRE vs. HitR
Our analysis revealed only a moderate correlation between TRE and HitR (Pearson $r = -0.49$). This discrepancy highlights that algorithms topping TRE-based leaderboards may not necessarily be the most clinically dependable tools, as they might be optimized for sub-resolution precision at the expense of overall robustness.
Discussion and Future Work
While HitR offers a more clinical perspective, it has limitations. Currently, ROIs are modeled as simple spheres, whereas anatomical uncertainty is often directional. Future iterations could incorporate non-spherical topologies. Additionally, the binary nature of HitR makes it non-differentiable, preventing its direct use as a loss function in deep learning. Developing "signed distance" variants could bridge this gap.
In conclusion, HitR shifts the focus from raw distance to clinical utility, providing a framework that better aligns algorithmic evaluation with the needs of medical practitioners.
Summary updated