Hyperspectral image segmentation: a preliminary study on the Oral and Dental Spectral Image Database (ODSI-DB)

Luis C. García-Peraza-Herrera, Conor C. Horgan, Sébastien Ourselin, Michael Ebner, Tom Vercauteren

TL;DR

This paper investigates the utility of hyperspectral imaging (HSI) for semantic segmentation of oral and dental tissues, comparing it against traditional RGB imaging. They found that while spatial information is the primary driver of segmentation performance, HSI offers a notable improvement, particularly for distinguishing subtle tissue variations.

What Problem Does This Paper Address?

Accurately identifying and segmenting different tissue types in the oral and dental region is crucial for surgical interventions and diagnostics. Traditional RGB imaging, mimicking human vision, often lacks the contrast and spectral detail needed to differentiate subtle pathological conditions or fine anatomical structures. Hyperspectral imaging (HSI) captures a much richer spectral signature of tissues, extending beyond the red, green, and blue bands. However, its application in medical imaging, especially for semantic segmentation, is still under-explored. This paper aims to quantify the performance gains of HSI over RGB for this task and understand the relative importance of spectral resolution and spatial context.

Methods

The core of this study involves training state-of-the-art deep learning segmentation models (specifically a 2D U-Net architecture) on different input data configurations:

  • RGB pixels (rgbpixel): Individual pixels from standard RGB images.
  • Hyperspectral pixels (spixel): Individual pixels from HSI, utilizing all available spectral bands.
  • RGB images (rgbimage): Full RGB images reconstructed from HSI.
  • Hyperspectral images (simage): Full HSI cubes.

A key methodological contribution is an improved RGB reconstruction method for hyperspectral images. The authors identified a yellow artifact in existing reconstructions from the Nuance EX camera (due to missing spectral information in the 400-450 nm range) and developed a novel color matching function (CMF) approach to compensate for these missing wavelengths, ensuring more realistic RGB outputs. They also linearly interpolated spectral bands to a common set of 170 for consistent processing across different cameras.

Datasets

The primary dataset used is the Oral and Dental Spectral Image Database (ODSI-DB). This dataset comprises 316 hyperspectral images (215 annotated) from 30 human subjects, with 35 distinct tissue classes. The images were captured using two different cameras: Nuance EX (51 spectral bands, 450-950 nm) and Specim IQ (204 spectral bands, 400-1000 nm). The authors provide their training code and models, and their improved RGB reconstruction code is also made available.

Key Insights and Ideas

  • Spatial Information Dominates: The most significant leap in segmentation performance comes from moving from pixel-wise analysis (rgbpixel/spixel) to image-based segmentation (rgbimage/simage). This highlights that contextual information within an image is paramount for accurate segmentation.
  • HSI Offers Subtle but Important Gains: While spatial context is king, HSI still provides a measurable improvement over RGB, especially when comparing pixel-wise inputs. This suggests HSI captures discriminative spectral features that are not present in RGB.
  • Improved RGB Reconstruction Matters: The authors' novel RGB reconstruction method effectively mitigates artifacts and ensures more consistent inputs, which is crucial for training models on datasets with mixed camera sources.
  • Specific Tissues Benefit Most: HSI shows a particular advantage in segmenting tissues like attached gingiva, where spectral information helps to better distinguish it from other structures.

Key Results

  • Image-based vs. Pixel-based: Image-based segmentation (rgbimage/simage) achieved significantly higher accuracies than pixel-based methods. For example, image-based RGB accuracy was 52.51%, while pixel-based RGB was 39.39%.
  • HSI vs. RGB (Image-based): When comparing image-based inputs, hyperspectral images (simage) achieved a balanced accuracy of 76.18%, compared to 74.39% for RGB images (rgbimage), a modest but consistent improvement.
  • HSI vs. RGB (Pixel-based): The difference was more pronounced at the pixel level. Hyperspectral pixels (spixel) achieved a balanced accuracy of 56.57%, while RGB pixels (rgbpixel) were at 55.46%. The image-based results showed a larger gap, with hyperspectral images outperforming RGB images by over 10 pp in accuracy.
  • Specific Class Improvements: For attached gingiva, hyperspectral information improved balanced accuracy from 63.09% (RGB) to 70.06% (hyperspectral).

Limitations and Future Work

  • Limitations: The authors acknowledge that their study uses a standard 2D U-Net, which might not fully exploit the spectral dimension of HSI. They also note that some classes in the ODSI-DB dataset are heavily underrepresented, making them difficult to evaluate.
  • Future Work:
    • Investigate if HSI can reduce the manual annotation effort required for medical imaging datasets.
  • Explore novel convolutional architectures specifically designed to leverage the spectral information in HSI, rather than simply treating it as an extended input to existing RGB models.
  • Further explore the potential of HSI for computer-assisted diagnosis and the development of quantitative biomarkers.

Summary updated

Abstract

Visual discrimination of clinical tissue types remains challenging, with traditional RGB imaging providing limited contrast for such tasks. Hyperspectral imaging (HSI) is a promising technology providing rich spectral information that can extend far beyond three-channel RGB imaging. Moreover, recently developed snapshot HSI cameras enable real-time imaging with significant potential for clinical applications. Despite this, the investigation into the relative performance of HSI over RGB imaging for semantic segmentation purposes has been limited, particularly in the context of medical imaging. Here we compare the performance of state-of-the-art deep learning image segmentation methods when trained on hyperspectral images, RGB images, hyperspectral pixels (minus spatial context), and RGB pixels (disregarding spatial context). To achieve this, we employ the recently released Oral and Dental Spectral Image Database (ODSI-DB), which consists of 215 manually segmented dental reflectance spectral images with 35 different classes across 30 human subjects. The recent development of snapshot HSI cameras has made real-time clinical HSI a distinct possibility, though successful application requires a comprehensive understanding of the additional information HSI offers. Our work highlights the relative importance of spectral resolution, spectral range, and spatial information to both guide the development of HSI cameras and inform future clinical HSI applications.
BibTeX
@inproceedings{garciaperazaherrera2023hyperspectral,
  author = {Luis C. García-Peraza-Herrera and Conor C. Horgan and Sébastien Ourselin and Michael Ebner and Tom Vercauteren},
  title = {{Hyperspectral image segmentation: a preliminary study on the Oral and Dental Spectral Image Database (ODSI-DB)}},
  booktitle = {Comput. Methods Biomech. Biomed. Eng. Imaging Vis},
  year = {2023},
  url = {https://openreview.net/forum?id=lfcP1r4CUy},
}
Figure