Coercive region-level registration for multi-modal images.

Yu-Hui Chen, Dennis L. Wei, Gregory E. Newstadt, Jeffrey P. Simmons, Alfred O. Hero Iii

TL;DR

This paper introduces a novel "coercive" method that simultaneously registers and segments multi-modal images by treating them as a unified problem. By operating at the region level, it avoids interpolation artifacts and leverages the strengths of different imaging modalities to achieve superior accuracy in material science applications.

What Problem Does This Paper Address?

Analyzing materials often requires combining information from different imaging techniques, such as Scanning Electron Microscopy (SEM) which can provide topographical (BSE) and crystallographic (EBSD) data. However, these multi-modal images can have different resolutions, distortions, and pixel value ranges, making it challenging to align them accurately (registration) and identify meaningful structures (segmentation). Traditional pixel-level registration methods often rely on interpolation, which can introduce smoothing artifacts and bias results. Independent segmentation of each modality can also fail due to low contrast in certain images or difficulties in matching segmented regions across modalities. This paper tackles the problem of performing registration and segmentation jointly and at a higher level of abstraction – the region level – to overcome these limitations.

Methods

The core of the proposed method is a "coercive" framework that simultaneously optimizes for both registration and segmentation. It operates by alternating between two steps:

  1. Region Splitting (Hypothesis Testing): The algorithm uses statistical hypothesis tests to determine if a current region in one modality should be split into two. This is crucial for refining segmentations and identifying boundaries that might be missed due to low contrast or misalignment. They develop tests for both univariate Gaussian (for BSE images) and von Mises-Fisher (for EBSD images, accounting for crystal symmetry) distributions. A second hypothesis test specifically differentiates between true missing boundaries and simple misalignment.
  2. Region Registration: Once potential splits are identified, the algorithm registers the boundaries of these regions. This is achieved by minimizing an objective function that balances the intra-modal consistency (how well pixel values within a region fit a statistical model) and an inter-modal energy term that penalizes topological differences between segmented regions in the two modalities. The optimization is performed using an alternating minimization approach.

The method is initialized by segmenting one image (preferably EBSD, which offers more accurate grain information) and then coarsely mapping this segmentation to the other modality using an affine transformation. The subsequent alternating steps refine both the segmentation and registration.

Datasets

The paper evaluates its approach on:

  • Simulated EBSD and BSE images: These are generated with controlled noise, distortions, and grain structures to create ground truth for boundary detection accuracy.
  • Real microscopy data (IN100 data set): This dataset consists of 170 slices of EBSD and BSE images of a Ni-base alloy, providing a realistic testbed for the algorithm's performance.

Key Insights and Ideas

  • Region-level processing is key: By avoiding pixel-level interpolation, the method sidesteps common artifacts and biases, leading to more faithful representations of material structures.
  • Joint registration and segmentation is more powerful: Treating these as coupled problems allows information from one modality to actively improve the segmentation and registration of the other, leading to a synergistic improvement.
  • Statistical models are crucial for robustness: Using statistical models (Gaussian for BSE, VMF for EBSD) allows the algorithm to robustly handle variations in pixel values within regions and to quantify uncertainty.
  • Hypothesis testing for boundary refinement: The proposed hypothesis tests are a novel way to dynamically refine segmentations by identifying and splitting regions that are statistically inconsistent, effectively detecting "missing" boundaries.
  • Coercive approach: The term "coercive" highlights how the algorithm forces consistency between the modalities at the region level, driving both registration and segmentation towards a unified solution.

Key Results

  • Superior Boundary Detection: On simulated data, the proposed coercive approach significantly outperforms independent segmentation and B-spline registration methods in terms of boundary overlapping rate, especially for smaller boundary widths.
  • Accurate Registration and Segmentation: For real microscopy data, the method successfully corrects misaligned boundaries from initial segmentations and accurately detects and locates missing boundaries within grain regions, as visualized in Figure 4.
  • Preservation of Region Correspondences: The region-level approach naturally preserves correspondences, which is beneficial for subsequent data fusion tasks.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed, region-based methods and iterative optimization can be computationally intensive.
  • Statistical Model Assumptions: The performance relies on the accuracy of the assumed statistical models (Gaussian, VMF). If the actual data deviates significantly, performance might be affected.
  • Future Work: The paper suggests that the framework is generalizable to other multi-modal image types and statistical models. Further exploration of different inter-modal energy functions and more complex statistical models could extend its applicability. The authors also mention that the boundary length penalty in the intra-modal energy was neglected for simplicity, implying its inclusion could be a future refinement.

Summary updated

Abstract

We propose a coercive approach to simultaneously register and segment multi-modal images which share similar spatial structure. Registration is done at the region level to facilitate data fusion while avoiding the need for interpolation. The algorithm performs alternating minimization of an objective function informed by statistical models for pixel values in different modalities. Hypothesis tests are developed to determine whether to refine segmentations by splitting regions. We demonstrate that our approach has significantly better performance than the state-of-the-art registration and segmentation methods on microscopy images.
BibTeX
@inproceedings{chen2015coercive,
  author = {Yu-Hui Chen and Dennis L. Wei and Gregory E. Newstadt and Jeffrey P. Simmons and Alfred O. Hero III},
  title = {{Coercive region-level registration for multi-modal images.}},
  booktitle = {ICIP},
  year = {2015},
  url = {https://openreview.net/forum?id=fK5vfTR9gUTL},
}
Figure