On the Sampling Strategy for Evaluation of Spectral-spatial Methods in Hyperspectral Image Classification

Jie Liang, Jun Zhou, Yuntao Qian, Lian Wen, Xiao Bai, Yongsheng Gao

Introduction

Hyperspectral image (HSI) classification has seen significant advancements with the integration of spectral-spatial methods. However, a critical flaw exists in the standard evaluation protocols used in the field. Many models that achieve near-perfect accuracy in laboratory settings often fail to generalize to real-world applications. This discrepancy is largely due to a "data leakage" trap inherent in traditional random sampling strategies.

The Problem: Spatial Overlap and the "Spatial Cheat"

Hyperspectral images are unique because they contain hundreds of spectral bands for every pixel, but labeled data is often scarce and difficult to obtain. To evaluate models, researchers typically take a single image and randomly split its pixels into training and testing sets.

When using spectral-spatial methods, which consider a neighborhood "window" (e.g., $3\times3$ or $5\times5$) around a pixel to capture texture and shape, this random split becomes problematic. Because classes in HSIs are spatially clustered, a training pixel and a testing pixel are frequently neighbors. Their spatial windows overlap significantly, meaning the model effectively "sees" the test data during training. This violation of the independence assumption between training and testing sets leads to inflated performance metrics.

Methodology

To address this bias, the paper proposes a Controlled Random Sampling Strategy. The goal is to ensure spatial independence by clustering training samples into local "islands" rather than a "salt-and-pepper" distribution across the image.

Figure 1: The general workflow for hyperspectral image classification, illustrating the parallel training and testing pipelines and the critical role of the sampling stage.

The strategy involves three main steps:

  1. Partitioning: Identifying unconnected regions for each class.
  2. Seed & Grow: Randomly selecting a "seed" pixel and using a region-growing algorithm to select adjacent pixels for training.
  3. Clustering: Concentrating samples to minimize the probability of window overlap between training and testing sets.

The authors also provide a theoretical foundation using Computational Learning Theory, specifically utilizing Rademacher complexity and $\beta$-mixing to explain how data dependence artificially tightens error bounds and inflates accuracy.

Datasets and Experimental Setup

The study evaluates the proposed strategy across five widely used hyperspectral benchmarks, representing diverse environments from wetlands to urban areas.

Figure 2: Original hyperspectral images (top) and their corresponding ground truth maps (bottom) for the five benchmark datasets: Botswana, Indian Pines, KSC, Pavia University, and Salinas.

Key Results and Discussion

The impact of switching from random to controlled sampling is stark. Using common spectral-spatial features like 3D Discrete Wavelet Transforms (3D-DWT), the authors observed significant accuracy drops. For example, on the Indian Pines dataset with a 5% training rate, accuracy plummeted from 88.1% under random sampling to 65.2% with controlled sampling.

This ~23% gap represents the "unearned" accuracy provided by data leakage. The lower score provides a much more honest reflection of how the model would perform on completely new, unseen imagery. This trend was consistent across all datasets and classifiers (SVM and Random Forest), proving that the bias is a systemic issue in HSI research.

Conclusion

While controlled sampling is a significant improvement, it presents a trade-off: clustered samples may not capture the full spectral variance of a class across an entire image. However, in the absence of multi-temporal benchmark datasets, the authors argue that controlled sampling is the best available compromise for rigorous HSI research. This work serves as a critical call for more objective evaluation standards in the remote sensing community.

Summary updated

BibTeX
@article{liang2017sampling,
  author = {Jie Liang and Jun Zhou and Yuntao Qian and Lian Wen and Xiao Bai and Yongsheng Gao},
  title = {{On the Sampling Strategy for Evaluation of Spectral-spatial Methods in Hyperspectral Image Classification}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2017},
  doi = {10.1109/tgrs.2016.2616489},
}
Figure