Needles & Haystacks: Dataset and Benchmark for Domain-Agnostic Image-Based Rigid Slice-to-Volume Registration

Anton Frolov, Florian Kleiner, Christiane Rößler, Volker RodehorstWACV 2025

TL;DR

This paper introduces "Needles & Haystacks," a novel dataset and benchmark for domain-agnostic slice-to-volume (S2V) registration, enabling the alignment of 2D slices into 3D volumes without prior domain knowledge. This is crucial for applications beyond medical imaging, such as materials science, where existing methods fall short due to the lack of specialized auxiliary information.

What Problem Does This Paper Address?

Slice-to-volume (S2V) registration is the task of aligning 2D tomographic slices into a 3D volume. While well-established in medical imaging, where anatomical knowledge (like landmarks or segmentation masks) aids the process, many other scientific fields face a more challenging "domain-agnostic" version. In materials science, geology, or archaeology, subjects lack predefined structures, making S2V inherently ill-posed due to symmetries, noise, and unstructured regions. Existing methods often require manual intervention or domain-specific information, which is unavailable in these contexts. This paper aims to bridge this gap by providing a comprehensive dataset and benchmark to drive research in robust, automated domain-agnostic S2V registration.

Methods

The authors tackle domain-agnostic S2V registration with a focus on rigid transformations. They propose a novel dataset, "Needles & Haystacks," created through a principled synthetic sampling strategy. This involves taking existing, openly licensed 3D volumetric data from diverse scientific domains (life sciences, materials characterization, paleo/archeo/anthropology) and generating a vast number of registration tasks with error-free ground truth. This synthetic approach ensures the dataset is openly licensed and promotes reproducibility.

To evaluate performance, they implement and test two main approaches:

  1. Optimization-based registration: This involves minimizing a dissimilarity criterion (like Mean Absolute Error, Mean Squared Error, or Zero-mean Normalized Cross-Correlation) between the transformed slice and the volume. They explore various classical optimization solvers (SLSQP, LBFGS, Subplex, COBYLA) and find that MAE combined with SLSQP offers the best performance, albeit with significant computational cost.
  2. Learned local features: Inspired by recent advances in image matching, they adapt the LoFTR (Local Feature Transformer) architecture for S2V. This "LoFTR-S2V" approach extracts features from both 2D and 3D data, performs coarse-level matching, and then refines the pose. They also incorporate robust registration techniques (like GC-RANSAC) and local optimization for refinement.

Datasets

The core contribution is the "Needles & Haystacks" dataset, which comprises 158,436 unique rigid S2V registration problems. This dataset is derived from 22 diverse sources of openly licensed 3D volumetric data, categorized into Life Sciences, Materials Characterization, and Paleo-, Archeo- & Anthropology. The dataset is split into training, validation, and testing sets. Additionally, they provide an online evaluation platform with 8,461 test problems for reproducible benchmarking of competing methods.

Key Insights and Ideas

  • Domain-agnostic S2V is significantly harder: The lack of prior domain knowledge and the inherent ambiguities in 2D-3D alignment make this problem substantially more challenging than its medical counterpart.
  • Synthetic data generation is key: Creating a large-scale, diverse, and ground-truth-rich dataset for domain-agnostic S2V is practically infeasible through manual collection. The authors demonstrate the effectiveness of a principled synthetic sampling approach, mirroring successful strategies in image matching.
  • Learned local features offer a promising path: While optimization-based methods can achieve reasonable accuracy, they are computationally expensive. Deep learning approaches like LoFTR-S2V show significant speedups while achieving comparable or better accuracy, especially after refinement.
  • Robustness is critical: The "Needles & Haystacks" dataset contains many challenging cases where simple optimization fails, highlighting the need for robust registration techniques, particularly when dealing with potential outliers from feature matching.

Key Results

  • The "Needles & Haystacks" dataset contains 158,436 registration tasks, with a train/validation/test split of 149,895 / 8,541 / 8,461 problems.
  • Optimization-based methods, even with extensive search (Nrand=2048), achieve a maximum mean Average Accuracy (mAA) of around 54.62% @ 20° angular error, indicating the dataset's difficulty.
  • LoFTR-S2V, a learned local feature approach, achieves comparable accuracy (52.58% mAA @ 20°) but is dramatically faster, taking approximately 0.297s per task compared to optimization methods that can take tens of seconds.
  • Refining LoFTR-S2V results with a single pass of local optimization (SLSQP) significantly improves accuracy, reaching 52.72% mAA @ 20° while still being much faster than pure optimization.

Limitations and Future Work

  • Rigid Registration Focus: The current work focuses solely on rigid S2V registration. Extending to non-rigid registration and cross-modal registration (e.g., aligning optical images with X-ray volumes) are identified as exciting future directions.
  • Computational Cost of Optimization: While optimization-based methods provide a baseline, their high computational cost limits their practical application for large-scale problems.
  • Dataset Representativeness: While diverse, the authors acknowledge that the dataset is a sample and further work could expand its scope.
  • Learned Method Generalization: While LoFTR-S2V shows promise, further research is needed to explore its generalization capabilities across even more diverse domains and modalities.

Summary updated

BibTeX
@inproceedings{frolov2025needles,
  author = {Anton Frolov and Florian Kleiner and Christiane Rößler and Volker Rodehorst},
  title = {{Needles \& Haystacks: Dataset and Benchmark for Domain-Agnostic Image-Based Rigid Slice-to-Volume Registration}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure