Abstract
With the increasing availability of new image registration approaches, an unbiased evaluation is becoming more needed so that clinicians can choose the most suitable approaches for their applications. Current evaluations typically use landmarks in manually annotated datasets. As a result, the quality of annotations is crucial for unbiased comparisons. Even though most data providers claim to have quality control over their datasets, an objective third-party screening can be reassuring for intended users. In this study, we use the variogram to screen the manually annotated landmarks in two datasets used to benchmark registration in image-guided neurosurgeries. The variogram provides an intuitive 2D representation of the spatial characteristics of annotated landmarks. Using variograms, we identified potentially problematic cases and had them examined by experienced radiologists. We found that (1) a small number of annotations may have fiducial localization errors; (2) the landmark distribution for some cases is not ideal to offer fair comparisons. If unresolved, both findings could incur bias in registration evaluation.
Introduction
In image-guided neurosurgery, image registration is the process of aligning different images—such as high-resolution preoperative MRI with real-time intraoperative ultrasound—to provide surgeons with precise spatial context. To evaluate the performance of registration algorithms, researchers typically rely on manually annotated landmarks (fiducials) as the ground truth.
However, this reliance assumes that these datasets are perfect "gold standards." In reality, two major issues can compromise evaluations:
- Fiducial Localization Error (FLE): Human annotators can make mistakes, placing landmarks incorrectly by several millimeters.
- Spatial Bias: If landmarks are clustered in specific areas, an algorithm might appear highly accurate in those regions while failing elsewhere, leading to misleading performance metrics.
This study asks whether we can objectively screen these public datasets for such biases using third-party tools.
Methods: Variogram-Based Screening
The authors propose the variogram—a tool from geostatistics—as an objective screening method. The underlying principle is that brain tissue deformation is generally smooth. Consequently, the displacement vectors of landmarks that are close to each other should be more similar than those of landmarks far apart.
For every pair of landmarks, the distance between them ($h$) and the squared difference of their displacement vectors ($\epsilon$) are calculated. These are then plotted to create a "variogram cloud." A healthy, consistent dataset should show a steady increasing trend. Deviations from this trend, such as local or global outliers, indicate potential annotation errors or poor spatial distribution.
Datasets
The researchers screened two high-quality public datasets used in neurosurgery:
- RESECT: 23 patients with brain tumors, featuring MRI and intraoperative ultrasound.
- BITE: 14 patients, including pre- and post-operative MRI and ultrasound. In total, over 700 landmark pairs were analyzed.
Key Results and Discussion
The variogram screening identified significant issues that could skew registration benchmarking.
Error Detection and Expert Validation
The tool flagged 29 landmark pairs as potential FLEs. To validate these findings, three neuro-radiologists performed a blind review of the flagged points. The landmarks identified as "certain" errors by the variogram received an average quality score of only 1.5 out of 4 from the experts, confirming the tool's accuracy in identifying human error.
Spatial Distribution and Benchmarking Impact
The screening also revealed 4 cases of landmark clustering and 11 cases of isolated landmarks. Such distributions can unfairly favor certain algorithms. For example, an algorithm might align inaccurate markers perfectly, resulting in a low error score that does not reflect real-world clinical performance. This "false" success can mislead clinicians when choosing tools for the operating room.
Conclusion
Public datasets, while essential, require objective screening to ensure unbiased evaluation. The variogram provides a powerful "fingerprint" of a dataset's spatial characteristics, making it easier to detect human error and spatial bias. The authors advocate for dataset providers to include spatial metadata and for the research community to move toward more robust evaluation criteria, such as surface-based alignment, to better support clinical decision-making.
Summary updated