TL;DR
This review paper provides a comprehensive overview of the burgeoning field of deep learning applied to medical hyperspectral imaging (MHSI). It highlights how combining the rich spectral and spatial information from HSI with the powerful pattern recognition capabilities of deep learning is revolutionizing disease diagnosis, detection, and segmentation, offering a path towards more accurate, non-invasive, and cost-effective medical imaging solutions.
What Problem Does This Paper Address?
Traditional medical imaging techniques like MRI and X-ray, while valuable, can be expensive, invasive, or even harmful. Hyperspectral imaging (HSI) offers a promising alternative by capturing detailed spectral information across numerous narrow bands, providing insights into tissue physiology and morphology that are invisible to conventional cameras. However, the high dimensionality of HSI data presents a significant challenge for analysis. This paper addresses the need for effective methods to process and interpret this complex data, specifically by reviewing how deep learning (DL) techniques can unlock the full potential of MHSI for medical applications, from early disease detection to surgical guidance.
Methods
The paper delves into the core components of MHSI and its integration with deep learning. It begins by explaining the principles of hyperspectral imaging, including various acquisition modes like whiskbroom, push broom, staring, and snapshot systems, and discusses their respective advantages and disadvantages. A significant portion is dedicated to image pre-processing techniques crucial for MHSI, such as normalization, smoothing/denoising, wave selection, and feature dimensionality reduction (e.g., PCA, LDA). The review then extensively covers the application of deep learning architectures, with a strong emphasis on Convolutional Neural Networks (CNNs) and their variants (e.g., 2D CNN, 3D CNN, U-Net, ResNet, DenseNet). It details how these DL models are used for classification, detection, and segmentation tasks within medical hyperspectral images, often leveraging dual-branch architectures to simultaneously extract spatial and spectral features for enhanced performance.
Datasets
The paper acknowledges the scarcity of publicly available datasets for MHSI research due to privacy and ethical concerns. However, it lists several relevant datasets that have been used in the literature:
- HSI Human Brain Database: Available from https://hsibraindatabase.iuma.ulpgc.es/
- MALDI rat liver anticancer drug spiked-in dataset (imzML): Available from https://www.ebi.ac.uk/pride/archive/projects/PXD016146
- The Hyperspectral SRS and Fluorescence data: Available from https://figshare.com/articles/dataset/hs_SRS_fluo_images_zip/13497138
- A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract: Data available from https://www.repository.cam.ac.uk/handle/1810/270691
- Parallel Implementations Assessment of a Spatial–Spectral Classifier for Hyperspectral Clinical Applications: Available from https://ieee-dataport.org/open-access/dataset-parallel-implementations-assessment-spatial-spectral-classifier-hyperspectral
- Microscopic Hyperspectral Choledoch Dataset: Available from https://www.kaggle.com/datasets/ethelzq/multidimensional-choledoch-database
- Multispectral Imaging Dataset of Colorectal tissue: Available from https://figshare.com/articles/figure/Multispectral_Imaging_Dataset_of_Colorectal_tissue/6224957/1
The authors also mention that some datasets are available upon request or from specific research groups.
Key Insights and Ideas
- HSI + DL is a Powerful Combination: The synergy between the rich spectral information from HSI and the advanced feature extraction capabilities of deep learning is proving to be a game-changer for medical diagnosis.
- Beyond RGB: HSI can reveal subtle spectral nuances indicative of disease that are completely missed by standard RGB imaging, offering a more comprehensive view of tissue health.
- Addressing High Dimensionality: Pre-processing techniques and band selection are critical for managing the "curse of dimensionality" inherent in HSI data, making it amenable to DL analysis.
- Dual-Branch Architectures Shine: Models that simultaneously process spatial and spectral information (dual-branch CNNs) tend to outperform those that treat them separately, leading to more accurate classifications.
- U-Net's Versatility: While initially for segmentation, U-Net and its variants are increasingly being adapted for classification and detection tasks in MHSI.
- The Need for Standardized Datasets: The lack of large, publicly available, and well-annotated MHSI datasets is a significant bottleneck for advancing the field.
Key Results
The review highlights numerous studies demonstrating the efficacy of DL in MHSI across various medical applications:
- Cancer Detection & Classification: High accuracies (often >90%) are reported for detecting and classifying various cancers, including stomach, brain, head and neck, skin, and colon cancers. For instance, one study achieved >96% accuracy, specificity, and sensitivity for gastric cancer classification.
- Tissue Segmentation: DL models, particularly U-Net architectures, show promising results in accurately segmenting tissues, aiding in surgical guidance.
- Cell Classification: CNNs have demonstrated superior performance over traditional methods like SVM in classifying blood cells.
- Non-invasive Diagnosis: The non-invasive nature of HSI, combined with DL, enables early detection and diagnosis, potentially reducing the need for invasive procedures.
Limitations and Future Work
The authors identify several key limitations and suggest future research directions:
- Limited Scope and Data: Current research is often narrow in scope, focusing on cell or tissue classification. There's a critical need for more diverse datasets, as existing ones are often scarce, small, and lack high-quality calibration, leading to overfitting risks. Data augmentation is a workaround but carries its own risks.
- Algorithm Development: While DL models are powerful, there's a continuous need to develop more efficient algorithms that can extract richer information from high spectral and spatial resolutions without losing detailed information.
- Hardware and Real-time Application: Hyperspectral acquisition devices are often large and cumbersome, limiting their application. The development of more compact, handheld devices and their integration with DL for real-time medical analysis is a crucial area for future work.
- Theoretical Foundation: While DL is rapidly advancing, its application in MHSI still lacks a robust theoretical foundation, necessitating more research to bridge the gap between theory and practice.
- Interdisciplinary Collaboration: Combining HSI with other biomedical imaging modalities and developing more comprehensive databases will be vital for advancing the field.
Summary updated