TL;DR
This paper introduces a novel workflow for precisely aligning 2D photographs of apple slices with their corresponding 3D CT scans. This is achieved through a joint 2D-3D image registration method that optimizes all slices of an apple simultaneously, leading to more accurate and robust alignment than slice-by-slice methods. The developed workflow enables the creation of large, high-quality datasets crucial for understanding and detecting internal fruit disorders.
What Problem Does This Paper Address?
Long-term storage of apples can lead to internal disorders like browning, making them unmarketable. While CT imaging shows promise for detecting these issues, understanding how different disorders manifest in CT scans requires correlating them with visual appearance. However, aligning 2D photographs of apple slices with their 3D CT counterparts is challenging due to variations in imaging and the need for precise registration. Manual alignment is tedious and error-prone, limiting dataset size and accuracy. This paper tackles the problem of creating accurate, large-scale datasets of paired apple slice photographs and CT scans to facilitate the study and detection of internal disorders.
Methods
The proposed workflow consists of four main steps:
- Data Acquisition: Apples are first CT scanned, then sliced into parallel sections, and each slice is photographed.
- Image Segmentation: The outer shape of the apple is segmented from both CT slices and photographs. For CT scans, this is done using thresholding and morphological operations. For photographs, a deep learning approach (Mixed-scale dense convolutional neural network - MSD-net) is employed due to the visual similarity between the apple's interior and background.
- Image Registration: This is the core innovation. A joint 2D-3D image registration method is used. Unlike traditional methods that register each slice independently, this approach optimizes the alignment of all slices from a single apple simultaneously. This leverages the fact that slices are parallel and equally spaced, allowing for shared transformation parameters (scaling, spacing, rotation, offset). The optimization is performed in two stages: an initialization step (matching area profiles) and a full joint optimization step using gradient descent with momentum. This joint optimization improves robustness and accuracy, especially when individual slices have limited distinguishing features.
- Validation: The registration accuracy is measured using the In-Plane Core Endpoint Distance (IPCED), calculated from manually annotated core points that were not used in the registration process. Segmentation error is also evaluated.
Datasets
The paper introduces a new dataset comprising 1347 slice photographs and their corresponding registered CT slices, acquired from 107 'Kanzi' apples stored under controlled atmosphere conditions for 8 months. This dataset is made publicly available for research. The segmentation model was trained using 175 manually segmented photos.
Key Insights and Ideas
- Joint 2D-3D Registration is Key: The most significant insight is that treating all slices of an apple as a single entity for registration, rather than individually, dramatically improves accuracy and robustness. This is because it allows the method to leverage shared information across slices and overcome limitations of individual slice data.
- Outer Shape is Sufficient for Registration: The registration method effectively uses only the outer shape of the apple for alignment. This is crucial because internal features can be inconsistent or difficult to segment, but the apple's silhouette is generally well-defined in both modalities.
- CT for Early Detection: The study found that some apples appearing healthy upon visual inspection showed dark regions in CT scans, suggesting CT's potential for detecting early-stage browning that is not yet visually apparent.
- Deep Learning for Photo Segmentation: The necessity of a deep learning approach for segmenting apple photographs highlights the challenges of visual data and the power of modern ML techniques for complex segmentation tasks.
Key Results
- The developed workflow achieved an average In-Plane Core Endpoint Distance (IPCED) of 1.47 ± 0.40 mm across all 107 apples.
- The joint registration method demonstrated superior performance compared to separate slice registration, showing a decrease in IPCED as the number of slices used in the joint optimization increased.
- The segmentation quality metrics (Accuracy, Precision, Recall) were high, with an average edge distance of 0.195 ± 0.104 mm for the combined segmentation instances.
- The study identified six apples where visual inspection indicated no browning, but CT scans revealed dark regions, suggesting potential for early detection of disorders.
Limitations and Future Work
- Manual Annotation for Validation: While the registration is automated, the IPCED metric still relies on manual annotation of the apple core, which can be time-consuming.
- Slice Parallelism Assumption: The joint registration method assumes slices are parallel. While slight deviations can be handled, significant deviations might impact accuracy.
- Future Work: The authors suggest that with minor modifications, the workflow could be extended to other fruits or MRI scans. They also highlight the potential for using the registered photo-CT pairs for more detailed disorder classification and the possibility of integrating multi-modality cost functions to potentially eliminate the segmentation step entirely. The dataset created is also a valuable resource for future research.
Summary updated