Deep Low-Shot Learning for Biological Image Classification and Visualization From Limited Training Samples

Lei Cai, Zhengyang Wang, Rob Kulathinal, Sudhir Kumar, Shuiwang JiIEEE TNNLS 2020 · DOI · OpenReview

TL;DR

This paper tackles the challenge of classifying biological images, specifically Drosophila embryo gene expression patterns, when only limited precisely labeled data is available. They propose a novel "deep two-step low-shot learning" framework that leverages both coarsely labeled large datasets and finely labeled small datasets to train accurate classification models. Beyond prediction, their method also provides interpretable saliency maps to visualize developmental landmarks, aiding biological research.

What Problem Does This Paper Address?

Biological image analysis, particularly for understanding gene interactions and regulation, often relies on images like in situ hybridization (ISH) patterns in developing organisms. To study these processes accurately, researchers need to compare images from the same developmental stage. However, precisely labeling these images with specific developmental stages is incredibly time-consuming and requires expert biological knowledge. This bottleneck severely limits the amount of training data available for computational models. Consequently, existing methods struggle to achieve high accuracy in precise stage classification, hindering biological discovery. Furthermore, biologists need ways to interpret why a model makes a certain prediction and to visualize key developmental features, which current methods often lack.

Methods

The core contribution is a deep two-step low-shot learning framework. This framework is designed to train deep learning models effectively even with scarce, precisely labeled data. It consists of two main stages:

  1. Data-Level Learning: This stage leverages a larger dataset with coarse labels (e.g., broad stage ranges) to train a deep classification network. The key idea is that the feature extraction part of this network, trained on abundant but less precise data, can provide a good initialization for the subsequent fine-grained classification. They use a deep residual network (ResNet) as their base model.
  2. Feature-Level Learning: This stage refines the model by introducing a regularization technique. It aims to ensure that features extracted from images belonging to the same fine-grained class are highly similar, while features from different classes are dissimilar. This is achieved by constructing "reference sets" of representative samples for each class and minimizing a "similarity loss" based on cosine similarity between the input sample's features and these reference features. This stage fine-tunes the model using the small, precisely labeled dataset.

For interpretation, they employ saliency maps. These maps highlight which pixels in an input image contribute most to the model's prediction. They then use these saliency maps as masks to generate "masked genomewide expression maps" (GEMs), which provide biologically meaningful visualizations of developmental landmarks.

Datasets

The primary dataset used is the Berkeley Drosophila Genome Project (BDGP) dataset. This dataset contains a large number of Drosophila ISH images.

  • Coarse Labels (Dt): Approximately 36,908 images with stage range labels (e.g., 1-3, 4-6).
  • Fine Labels (Ds): Approximately 3,474 images with precise stage labels (e.g., specific stages from 4 to 17).

The paper notes that stages 1-3 are less biologically meaningful for precise prediction, so they are excluded from the fine-labeled dataset for this task.

Key Insights and Ideas

  • Leveraging Coarse Labels is Crucial: The paper demonstrates that even imperfect, large-scale annotations can be invaluable for initializing deep learning models when fine-grained labels are scarce. This is a significant departure from methods that solely rely on the limited fine-labeled data.
  • Feature Similarity as Regularization: The feature-level learning step, by explicitly enforcing feature similarity within classes and dissimilarity between classes, acts as a powerful regularization technique. This helps the model generalize better from limited examples.
  • Interpretability Enhances Biological Utility: The integration of saliency maps and masked GEMs transforms the model from a black box into a tool that can actively aid biological discovery by highlighting key features and developmental trends.
  • Generalizability: The authors suggest their framework is broadly applicable to other biological image classification tasks with limited training data, not just Drosophila ISH.

Key Results

  • Improved Accuracy: The proposed "Data&Feature-level" deep low-shot learning approach achieved the highest precise stage prediction accuracy, outperforming the baseline Gabor filter method and a "Plain ResNet" trained directly on the small dataset. For instance, in Table II, the average accuracy for the "Data&Feature-level" model reached 83.92%, compared to 80.97% for the Gabor filter baseline and 77.28% for the Plain ResNet.
  • Reduced Misclassifications: The deep learning models trained with their framework showed fewer misclassifications into incorrect broad stage ranges compared to the baseline.
  • Biologically Meaningful Visualizations: The generated masked GEMs, informed by saliency maps, provided more detailed and biologically relevant visualizations of gene expression patterns and developmental landmarks, consistent with established developmental timelines.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method significantly improves performance, the accuracy for some individual stages (e.g., Stage 9 in Table II) is still not perfect, indicating room for further improvement. The reliance on a threshold for constructing reference sets in feature-level learning might also be a point of sensitivity.
  • Future Work: The authors anticipate their methods to be easily generalizable to other biological image classification tasks with small training datasets. They also highlight the utility of their interpretation methods for identifying and visualizing developmental landmarks, suggesting further exploration in this area.

Summary updated

Abstract

Predictive modeling is useful but very challenging in biological image analysis due to the high cost of obtaining and labeling training data. For example, in the study of gene interaction and regulation in Drosophila embryogenesis, the analysis is most biologically meaningful when in situ hybridization (ISH) gene expression pattern images from the same developmental stage are compared. However, labeling training data with precise stages is very time-consuming even for developmental biologists. Thus, a critical challenge is how to build accurate computational models for precise developmental stage classification from limited training samples. In addition, identification and visualization of developmental landmarks are required to enable biologists to interpret prediction results and calibrate models. To address these challenges, we propose a deep two-step low-shot learning framework to accurately classify ISH images using limited training images. Specifically, to enable accurate model training on limited training samples, we formulate the task as a deep low-shot learning problem and develop a novel two-step learning approach, including data-level learning and feature-level learning. We use a deep residual network as our base model and achieve improved performance in the precise stage prediction task of ISH images. Furthermore, the deep model can be interpreted by computing saliency maps, which consists of pixel-wise contributions of an image to its prediction result. In our task, saliency maps are used to assist the identification and visualization of developmental landmarks. Our experimental results show that the proposed model can not only make accurate predictions but also yield biologically meaningful interpretations. We anticipate our methods to be easily generalizable to other biological image classification tasks with small training datasets. Our open-source code is available at https://github.com/divelab/lsl-fly .
BibTeX
@article{cai2020deep,
  author = {Lei Cai and Zhengyang Wang and Rob Kulathinal and Sudhir Kumar and Shuiwang Ji},
  title = {{Deep Low-Shot Learning for Biological Image Classification and Visualization From Limited Training Samples}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2020},
  doi = {10.1109/tnnls.2021.3106831},
}
Figure