Hyperspectral Image Classification Based on Parameter-Optimized 3D-CNNs Combined with Transfer Learning and Virtual Samples

Xuefeng Liu, Qiaoqiao Sun, Yue Meng, Min Fu, Salah BourennaneRemote 2018 · DOI

TL;DR

This paper introduces a novel approach for hyperspectral image (HSI) classification by combining parameter-optimized 3D-CNNs with transfer learning and virtual samples. This method effectively addresses the common challenge of limited labeled training data in HSIs, leading to significantly improved classification accuracy.

What Problem Does This Paper Address?

Hyperspectral images (HSIs) contain rich spectral information across hundreds of bands, making them valuable for detailed land cover classification. Three-dimensional Convolutional Neural Networks (3D-CNNs) are well-suited for HSI analysis as they can capture both spatial and spectral features simultaneously. However, a major bottleneck in applying deep learning models like 3D-CNNs to HSIs is the scarcity of labeled training samples. Acquiring and annotating large HSI datasets is a laborious and expensive process. This lack of sufficient training data can lead to poor model performance and overfitting. The paper aims to overcome this challenge by developing a method that leverages existing knowledge and synthetically generated data to boost classification accuracy.

Methods

The proposed method, termed PO-3DCNN-TV, integrates several key components:

  • Parameter Optimization (PO-3DCNN): The authors first optimize the hyperparameters of a 3D-CNN for the target HSI. This is done using a "single variable principle," where each parameter is adjusted individually while others remain fixed, and the optimal value is selected based on the Overall Accuracy (OA). Dropout is also incorporated during this phase to mitigate overfitting.
  • Transfer Learning (TL): To address the limited samples, knowledge from a well-trained 3D-CNN on a "source" HSI (with abundant samples and a similar feature space) is transferred to the target HSI. Specifically, the weights of the convolutional and pooling layers from the pre-trained source model are used to initialize the corresponding layers of the parameter-optimized target 3D-CNN. This leverages existing learned features.
  • Virtual Samples (VS): To further augment the training data, virtual samples are generated from the original samples of the target HSI. These synthetic samples are created by adding a scaled version of the original sample and Gaussian noise, simulating environmental interference. This increases the effective size of the training dataset.

The final model combines these elements: a 3D-CNN is first optimized for the target data, then its initial weights are informed by transfer learning from a source dataset, and finally, it is trained using a mix of original and virtual samples from the target data.

Datasets

The experiments were conducted using two real-world hyperspectral datasets:

  • University of Pavia (PaviaU) HSI: This dataset was used as the target data for classification. A subset, "part-PaviaU," was selected for the experiments, containing 9 land-cover classes.
  • Center of Pavia (PaviaC) HSI: This dataset was used as the source data for transfer learning. It has a similar spectral space to PaviaU but was used to provide abundant training samples.

The paper notes that both datasets were acquired by the ROSIS sensor. The number of spectral bands was reduced to 103 for PaviaU and 102 for PaviaC.

Key Insights and Ideas

  • Synergy of Techniques: The core insight is that combining parameter optimization, transfer learning, and virtual samples creates a powerful ensemble for HSI classification, outperforming individual techniques.
  • Parameter Optimization Matters: Even with advanced architectures like 3D-CNNs, careful tuning of parameters is crucial for achieving optimal performance, especially when dealing with specific datasets.
  • Transfer Learning as a Bridge: Transfer learning effectively bridges the gap between datasets with varying sample availability, allowing models to benefit from knowledge learned on data-rich sources.
  • Virtual Samples as Data Augmentation: Virtual samples provide a practical way to artificially increase the training dataset size, mitigating the negative impact of limited real-world annotations.
  • 3D-CNNs Excel at Spatial-Spectral Fusion: The comparison with 2D-CNNs highlights the inherent advantage of 3D-CNNs in capturing the spatio-spectral nature of HSI data.

Key Results

The proposed PO-3DCNN-TV method achieved the highest classification accuracy among all tested methods on the part-PaviaU HSI dataset.

  • Overall Accuracy (OA): The PO-3DCNN-TV method achieved an OA of 0.9962.
  • Comparison: This significantly outperformed SVM (0.9231), DBN (0.9505), PO-2DCNN (0.9764), PO-3DCNN (0.9891), PO-3DCNN-TL (0.9938), and PO-3DCNN-VS (0.9947).
  • Class-wise Performance: The method also showed excellent performance across most individual classes, with several achieving perfect or near-perfect accuracy (e.g., Asphalt, Meadows, Gravel, Bare Soil, Bitumen, Bricks all reaching 1 or 0.99+).
  • Qualitative Results: Visualizations (Figure 10) show that the proposed method produces much cleaner and more accurate classification maps with fewer misclassified pixels compared to baseline methods.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the performance of transfer learning relies on the availability of a suitable "source" HSI with sufficient samples and a similar feature space. If such a source dataset is absent, this component cannot be utilized.
  • The parameter optimization process, while effective, is described as being done "according to the single variable principle," which can be computationally intensive.
  • Future Work:
    • The authors suggest investigating the use of residual networks to further address potential overfitting issues in CNNs.
  • Further research could explore more advanced techniques for parameter optimization and transfer learning, potentially reducing computational costs or improving robustness to domain shifts.

Summary updated

Abstract

Recent research has shown that spatial-spectral information can help to improve the classification of hyperspectral images (HSIs). Therefore, three-dimensional convolutional neural networks (3D-CNNs) have been applied to HSI classification. However, a lack of HSI training samples restricts the performance of 3D-CNNs. To solve this problem and improve the classification, an improved method based on 3D-CNNs combined with parameter optimization, transfer learning, and virtual samples is proposed in this paper. Firstly, to optimize the network performance, the parameters of the 3D-CNN of the HSI to be classified (target data) are adjusted according to the single variable principle. Secondly, in order to relieve the problem caused by insufficient samples, the weights in the bottom layers of the parameter-optimized 3D-CNN of the target data can be transferred from another well trained 3D-CNN by a HSI (source data) with enough samples and the same feature space as the target data. Then, some virtual samples can be generated from the original samples of the target data to further alleviate the lack of HSI training samples. Finally, the parameter-optimized 3D-CNN with transfer learning can be trained by the training samples consisting of the virtual and the original samples. Experimental results on real-world hyperspectral satellite images have shown that the proposed method has great potential prospects in HSI classification.
BibTeX
@inproceedings{liu2018hyperspectral,
  author = {Xuefeng Liu and Qiaoqiao Sun and Yue Meng and Min Fu and Salah Bourennane},
  title = {{Hyperspectral Image Classification Based on Parameter-Optimized 3D-CNNs Combined with Transfer Learning and Virtual Samples}},
  booktitle = {Remote},
  year = {2018},
  doi = {10.3390/rs10091425},
}
Figure