Active Learning Plus Deep Learning Can Establish Cost-Effective and Robust Model for Multichannel Image: A Case on Hyperspectral Image Classification

Fangyu Shi, Zhaodi Wang, Menghan Hu, Guangtao Zhai

TL;DR

This paper introduces a cost-effective framework for multichannel image classification by combining active learning with deep learning. By intelligently selecting informative samples for manual annotation and leveraging data augmentation through an "image pool," the approach significantly reduces labeling effort while maintaining high classification accuracy, particularly for challenging hyperspectral images in agricultural applications.

What Problem Does This Paper Address?

Deep learning models have revolutionized image classification, but their reliance on massive, meticulously labeled datasets presents a significant bottleneck. This is especially true in specialized fields like agricultural and biological engineering, where obtaining accurate ground truth is not only time-consuming and expensive but also requires domain-specific expertise. Many of these applications involve multichannel images (e.g., hyperspectral images, CT scans), which further complicate the labeling process. The core challenge this paper tackles is how to achieve high classification performance with substantially less manual annotation effort, making advanced deep learning models more accessible and practical for these domains.

Methods

The authors propose a framework that integrates active learning with deep learning for multichannel image classification. The core idea is to iteratively train a deep learning model (specifically, a Residual Network or ResNet) while intelligently selecting the most informative unlabeled samples for manual annotation.

Here's a breakdown of their approach:

  • Active Learning Criteria: Three established active learning strategies are employed to select samples:
  • Least Confidence: Selects samples where the model is least confident about its prediction (i.e., the highest predicted probability is low).
  • Margin Sampling: Selects samples where the difference between the top two predicted class probabilities is smallest, indicating ambiguity.
  • Entropy: Selects samples with the highest information entropy, reflecting the most uncertainty across all possible classes.
  • Deep Learning Model: A Residual Network (ResNet) is used as the backbone for classification. This choice is motivated by ResNet's ability to handle deep architectures effectively and its proven performance in image recognition tasks.
  • "Image Pool" for Data Augmentation: A novel aspect is the introduction of an "image pool." When an image is manually annotated, all its associated images generated through data augmentation (e.g., flips, rotations) are added to this pool. In subsequent iterations, instead of randomly selecting new images for manual annotation, a fixed number of images are randomly drawn from this pool to serve as the "manually labeled" data. This strategy capitalizes on the fact that augmented images of the same original sample share the same label, effectively reducing the need for new manual annotations.
  • Pseudo-Labeling: The framework also incorporates pseudo-labeling, where the model's confident predictions on unlabeled data are used as training labels in later stages. The proportion of pseudo-labeled data increases over training iterations.

The overall process involves iteratively selecting samples based on active learning criteria, augmenting them, adding them to the image pool, and then using a combination of manually annotated and pseudo-labeled data to retrain the CNN.

Datasets

The primary dataset used for evaluation is a collection of hyperspectral transmittance images of blueberries.

  • Source: Collected from Frutera S.A., Chile.
  • Content: 575 blueberries, comprising 304 sound samples and 253 damaged samples.
  • Ground Truth: Determined by cutting the blueberries equatorially, as internal damage is not visible externally. Damage is classified if more than 25% of the cut surface shows damage.
  • Image Characteristics: Hyperspectral image cubes with dimensions varying from 100 to 130 pixels in width/height and 1002 spectral channels (wavelengths from 328.82 nm to 1113.54 nm).
  • Data Split: 80% of samples were randomly selected for training, and the remaining 20% for testing.

The paper does not introduce a new benchmark dataset but rather uses this existing dataset to demonstrate the effectiveness of their proposed framework.

Key Insights and Ideas

  • Active Learning is Key for Domain-Specific Data: The paper strongly validates the utility of active learning in domains where manual annotation is costly and expertise-dependent. It shows that intelligent selection can drastically reduce the annotation burden without sacrificing accuracy.
  • The "Image Pool" is a Game-Changer for Data Augmentation: The most significant insight is how effectively the "image pool" leverages data augmentation. By treating augmented versions of a single manually labeled image as if they were also manually labeled (drawn from the pool), the framework dramatically cuts down the number of unique images requiring human annotation. This is a clever way to maximize the value of each annotation.
  • Entropy as the Superior Active Learning Criterion: Across both algorithms, the entropy-based active learning criterion consistently achieved the highest accuracy with the fewest manually annotated samples, suggesting it's the most effective at identifying truly informative data points for this task.
  • Deep Learning + Active Learning Outperforms Baseline: The combined approach not only matches but often surpasses the performance of a fully manually annotated baseline model, especially when using the image pool strategy. This highlights the power of combining intelligent data selection with powerful deep learning architectures.

Key Results

  • Significant Reduction in Annotation Effort:
    • Algorithm 1 (Active Learning without Image Pool): Achieved performance comparable to the baseline (fully annotated) model by manually annotating 85% of the training set. Peak accuracy was 0.973.
  • Algorithm 2 (Active Learning with Image Pool): Achieved the baseline model's accuracy by manually annotating only 33% of the training set. The peak accuracy reached 0.964 with 35.9% manual annotation.
  • Highest Accuracy Achieved: The combination of Algorithm 2 and the entropy-based active learning criterion achieved an accuracy of 0.991 with only 41.5% of the training set manually annotated.
  • Comparison of Criteria: For Algorithm 2, entropy achieved the highest peak accuracy (0.991) with 41.5% annotation, followed by margin sampling (0.973 with 42.6%) and least confidence (0.964 with 35.9%).
  • Random Selection is Inferior: Replacing active learning with random selection for annotation resulted in lower peak accuracy, even with more samples manually annotated, underscoring the value of the active learning strategies.

Limitations and Future Work

  • Limitations:
    • The paper focuses on a specific type of multichannel image (hyperspectral transmittance images of blueberries). While the framework is generalizable, its performance might vary on other types of multichannel data or different agricultural/biological applications.
  • The "image pool" strategy relies heavily on the effectiveness of data augmentation. If augmentation techniques are not well-suited to the specific data, the pool's utility might be diminished.
  • The paper doesn't explicitly detail the computational cost increase due to the iterative nature of active learning and the management of the image pool, though it implies a reduction in overall labeling cost.
  • Future Work:
    • The authors suggest that the proposed framework can be applied to other multichannel image classification tasks in agricultural and biological engineering.
  • Further exploration of different active learning criteria or hybrid approaches could be beneficial.
  • Investigating the optimal size and management strategy for the "image pool" in various scenarios could be a direction for future research.

Summary updated

Abstract

Relying on large scale labeled datasets, deep learning has achieved good performance in image classification tasks. In agricultural and biological engineering, image annotation is time-consuming and expensive. It also requires annotators to have technical skills in specific areas. Obtaining the ground truth is difficult because natural images are expensive. In addition, images in these areas are usually stored as multichannel images, such as computed tomography (CT) images, magnetic resonance images (MRI), and hyperspectral images (HSI). In this paper, we present a framework using active learning and deep learning for multichannel image classification. We use three active learning algorithms, including least confidence, margin sampling, and entropy, as the selection criteria. Based on this framework, we further introduce an “image pool” to make full advantage of images generated by data augmentation. To prove the availability of the proposed framework, we present a case study on agricultural hyperspectral image classification. The results show that the proposed framework achieves better performance compared with the deep learning model. Manual annotation of all the training sets achieves an encouraging accuracy. In comparison, using active learning algorithm of entropy and image pool achieves a similar accuracy with only part of the whole training set manually annotated. In practical application, the proposed framework can remarkably reduce labeling effort during the model development and upadting processes, and can be applied to multichannel image classification in agricultural and biological engineering.
BibTeX
@inproceedings{shi2020active,
  author = {Fangyu Shi and Zhaodi Wang and Menghan Hu and Guangtao Zhai},
  title = {{Active Learning Plus Deep Learning Can Establish Cost-Effective and Robust Model for Multichannel Image: A Case on Hyperspectral Image Classification}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=auGisGg6TCO},
}
Figure