Uncertainty-Aware Annotation Protocol to Evaluate Deformable Registration Algorithms

Loïc Peter, Daniel C. Alexander, Caroline Magnain, Juan Eugenio Iglesias

Introduction

Deformable image registration (DIR)—the non-rigid alignment of two or more images—is a fundamental task in medical imaging, essential for applications ranging from longitudinal study analysis to image-guided surgery. Despite its importance, validating DIR algorithms remains a significant hurdle. Ideally, evaluation would involve comparing an algorithm's output to a known "ground truth" transformation. However, in real-world clinical scans, such ground truth is rarely available.

Traditionally, researchers have relied on manual landmarking, where experts identify corresponding points in image pairs. This approach is limited by three major factors: it is incredibly tedious to annotate enough points for dense evaluation; it is subjective, as experts often choose "easy" landmarks while ignoring ambiguous regions where registration is most likely to fail; and it ignores inherent spatial uncertainty. Traditional protocols force a single-point choice even when a landmark might realistically lie anywhere along an edge or within a small region.

A Principled, Uncertainty-Aware Framework

This paper introduces a principled protocol for creating "gold standard" landmark correspondences that explicitly accounts for these challenges. By leveraging Gaussian Processes (GPs) and active learning, the framework suggests the most informative locations for human annotation and incorporates spatial uncertainty directly into the evaluation process.

Methods

The core of the proposed approach is modeling the underlying transformation field as a Gaussian Process. This probabilistic framework allows for three key innovations:

  1. Active Learning for Efficient Annotation: Rather than requiring users to manually search for landmarks, the system iteratively suggests the most informative locations to annotate. Using an entropy-based criterion, it identifies regions that will most effectively reduce the overall uncertainty of the transformation model. This ensures a more representative gold standard with significantly fewer manual clicks.
  2. Uncertainty-Aware Annotations: The protocol moves beyond simple point-based landmarks. Users can specify an elliptic confidence region (anisotropic uncertainty) to indicate that a landmark lies somewhere along a specific anatomical boundary rather than at a single pixel.
Standard point-based landmarking often ignores spatial ambiguity.
The proposed protocol allows for elliptic confidence regions to capture anisotropic uncertainty, better reflecting human expert confidence.
  1. Probabilistic Evaluation: Instead of measuring error only at annotated points, the framework uses the GP to estimate registration error across the entire target domain. This provides a holistic view of algorithm performance, weighted by the model's confidence in its own ground-truth estimate.

Experimental Validation

The framework was validated across four diverse datasets, including synthetic 3D brain MRIs with known deformations, 2D multi-modal histology sections (CIMA), high-density brain tissue annotations, and 3D chest CT scans (COPDgene) with multi-annotator data.

Key Results

  • Superior Algorithm Ranking: The proposed method was significantly more effective at correctly ranking different registration algorithms compared to standard landmark-based metrics.
  • Increased Efficiency: In histology experiments, the active learning strategy reached stable evaluation scores with roughly half the number of landmarks required by traditional distance-based heuristics.
  • Robustness to Noise: By merging multiple expert annotations into the uncertainty-aware framework, the system produced a more accurate assessment of "true" registration error than using any single expert's points alone.

Conclusion and Future Work

By treating registration evaluation as a probabilistic problem, this framework provides a more robust and efficient way to assess DIR algorithms. It successfully captures the "I don't know" factor inherent in human landmarking, preventing noisy, forced-choice annotations from corrupting the evaluation.

Future work may involve incorporating image-derived information, such as tissue segmentation or edge maps, directly into the GP kernel to further refine anatomical constraints. While computationally more intensive than simple point-matching, the provided iterative implementation maintains the interactivity necessary for practical clinical use.

Summary updated

Abstract

Landmark correspondences are a widely used type of gold standard in image registration. However, the manual placement of corresponding points is subject to high inter-user variability in the chosen annotated locations and in the interpretation of visual ambiguities. In this paper, we introduce a principled strategy for the construction of a gold standard in deformable registration. Our framework: (i) iteratively suggests the most informative location to annotate next, taking into account its redundancy with previous annotations; (ii) extends traditional pointwise annotations by accounting for the spatial uncertainty of each annotation, which can either be directly specified by the user, or aggregated from pointwise annotations from multiple experts; and (iii) naturally provides a new strategy for the evaluation of deformable registration algorithms. Our approach is validated on four different registration tasks. The experimental results show the efficacy of suggesting annotations according to their informativeness, and an improved capacity to assess the quality of the outputs of registration algorithms. In addition, our approach yields, from sparse annotations only, a dense visualization of the errors made by a registration method. The source code of our approach supporting both 2D and 3D data is publicly available at https://github.com/LoicPeter/evaluation-deformable-registration.
BibTeX
@article{peter2021uncertainty,
  author = {Loïc Peter and Daniel C. Alexander and Caroline Magnain and Juan Eugenio Iglesias},
  title = {{Uncertainty-Aware Annotation Protocol to Evaluate Deformable Registration Algorithms}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2021},
  doi = {10.1109/tmi.2021.3070842},
}
Figure