Joint 2D to 3D image registration workflow for comparing multiple slice photographs and CT scans of apple fruit with internal disorders

Dirk Elias Schut, Rachael Maree Wood, Anna Katharina Trull, R.e. Schouten, Robert van Liere, Tristan van Leeuwen, Kees Joost Batenburg

TL;DR

This paper introduces a novel workflow for precisely aligning 2D photographs of apple slices with their corresponding 3D CT scans. This is achieved through a joint 2D-3D image registration method that optimizes all slices of an apple simultaneously, leading to more accurate and robust alignment than slice-by-slice methods. The developed workflow enables the creation of large, high-quality datasets crucial for understanding and detecting internal fruit disorders.

What Problem Does This Paper Address?

Long-term storage of apples can lead to internal disorders like browning, making them unmarketable. While CT imaging shows promise for detecting these issues, understanding how different disorders manifest in CT scans requires correlating them with visual appearance. However, aligning 2D photographs of apple slices with their 3D CT counterparts is challenging due to variations in imaging and the need for precise registration. Manual alignment is tedious and error-prone, limiting dataset size and accuracy. This paper tackles the problem of creating accurate, large-scale datasets of paired apple slice photographs and CT scans to facilitate the study and detection of internal disorders.

Methods

The proposed workflow consists of four main steps:

  1. Data Acquisition: Apples are first CT scanned, then sliced into parallel sections, and each slice is photographed.
  2. Image Segmentation: The outer shape of the apple is segmented from both CT slices and photographs. For CT scans, this is done using thresholding and morphological operations. For photographs, a deep learning approach (Mixed-scale dense convolutional neural network - MSD-net) is employed due to the visual similarity between the apple's interior and background.
  3. Image Registration: This is the core innovation. A joint 2D-3D image registration method is used. Unlike traditional methods that register each slice independently, this approach optimizes the alignment of all slices from a single apple simultaneously. This leverages the fact that slices are parallel and equally spaced, allowing for shared transformation parameters (scaling, spacing, rotation, offset). The optimization is performed in two stages: an initialization step (matching area profiles) and a full joint optimization step using gradient descent with momentum. This joint optimization improves robustness and accuracy, especially when individual slices have limited distinguishing features.
  4. Validation: The registration accuracy is measured using the In-Plane Core Endpoint Distance (IPCED), calculated from manually annotated core points that were not used in the registration process. Segmentation error is also evaluated.

Datasets

The paper introduces a new dataset comprising 1347 slice photographs and their corresponding registered CT slices, acquired from 107 'Kanzi' apples stored under controlled atmosphere conditions for 8 months. This dataset is made publicly available for research. The segmentation model was trained using 175 manually segmented photos.

Key Insights and Ideas

  • Joint 2D-3D Registration is Key: The most significant insight is that treating all slices of an apple as a single entity for registration, rather than individually, dramatically improves accuracy and robustness. This is because it allows the method to leverage shared information across slices and overcome limitations of individual slice data.
  • Outer Shape is Sufficient for Registration: The registration method effectively uses only the outer shape of the apple for alignment. This is crucial because internal features can be inconsistent or difficult to segment, but the apple's silhouette is generally well-defined in both modalities.
  • CT for Early Detection: The study found that some apples appearing healthy upon visual inspection showed dark regions in CT scans, suggesting CT's potential for detecting early-stage browning that is not yet visually apparent.
  • Deep Learning for Photo Segmentation: The necessity of a deep learning approach for segmenting apple photographs highlights the challenges of visual data and the power of modern ML techniques for complex segmentation tasks.

Key Results

  • The developed workflow achieved an average In-Plane Core Endpoint Distance (IPCED) of 1.47 ± 0.40 mm across all 107 apples.
  • The joint registration method demonstrated superior performance compared to separate slice registration, showing a decrease in IPCED as the number of slices used in the joint optimization increased.
  • The segmentation quality metrics (Accuracy, Precision, Recall) were high, with an average edge distance of 0.195 ± 0.104 mm for the combined segmentation instances.
  • The study identified six apples where visual inspection indicated no browning, but CT scans revealed dark regions, suggesting potential for early detection of disorders.

Limitations and Future Work

  • Manual Annotation for Validation: While the registration is automated, the IPCED metric still relies on manual annotation of the apple core, which can be time-consuming.
  • Slice Parallelism Assumption: The joint registration method assumes slices are parallel. While slight deviations can be handled, significant deviations might impact accuracy.
  • Future Work: The authors suggest that with minor modifications, the workflow could be extended to other fruits or MRI scans. They also highlight the potential for using the registered photo-CT pairs for more detailed disorder classification and the possibility of integrating multi-modality cost functions to potentially eliminate the segmentation step entirely. The dataset created is also a valuable resource for future research.

Summary updated

Abstract

A large percentage of apples are affected by internal disorders after long-term storage, which makes them unacceptable in the supply chain. CT imaging is a promising technique for in-line detection of these disorders. Therefore, it is crucial to understand how different disorders affect the image features that can be observed in CT scans. This paper presents a workflow for creating datasets of image pairs of photographs of apple slices and their corresponding CT slices. By having CT and photographic images of the same part of the apple, the complementary information in both images can be used to study the processes underlying internal disorders and how internal disorders can be measured in CT images. The workflow includes data acquisition, image segmentation, image registration, and validation methods. The image registration method aligns all available slices of an apple within a single optimization problem, assuming that the slices are parallel. This method outperformed optimizing the alignment separately for each slice. The workflow was applied to create a dataset of 1347 slice photographs and their corresponding CT slices. The dataset was acquired from 107 ‘Kanzi’ apples that had been stored in controlled atmosphere (CA) storage for 8 months. In this dataset, the distance between annotations in the slice photograph and the matching CT slice was, on average, 1.47 ± 0.40 mm. Our workflow allows collecting large datasets of accurately aligned photo-CT image pairs, which can help distinguish internal disorders with a similar appearance on CT. With slight modifications, a similar workflow can be applied to other fruits or MRI instead of CT scans.
BibTeX
@inproceedings{schut2024joint,
  author = {Dirk Elias Schut and Rachael Maree Wood and Anna Katharina Trull and R.E. Schouten and Robert van Liere and Tristan van Leeuwen and Kees Joost Batenburg},
  title = {{Joint 2D to 3D image registration workflow for comparing multiple slice photographs and CT scans of apple fruit with internal disorders}},
  booktitle = {Postharvest},
  year = {2024},
  doi = {10.1016/j.postharvbio.2024.112814},
}

Loading related content...

Figure