Leukocyte Classification using Multimodal Architecture Enhanced by Knowledge Distillation

Litao Yang, Deval Mehta, Dwarikanath Mahapatra, Zongyuan Ge

Abstract

Recently, a lot of automated white blood cells (WBC) or leukocyte classification techniques have been developed. However, all of these methods only utilize a single modality microscopic image i.e. either blood smear or fluorescence based, thus missing the potential of a better learning from multimodal images. In this work, we develop an efficient multimodal architecture based on a first of its kind multimodal WBC dataset for the task of WBC classification. Specifically, our proposed idea is developed in two steps - 1) First, we learn modality specific independent subnetworks inside a single network only; 2) We further enhance the learning capability of the independent subnetworks by distilling knowledge from high complexity independent teacher networks. With this, our proposed framework can achieve a high performance while maintaining low complexity for a multimodal dataset. Our unique contribution is two-fold - 1) We present a first of its kind multimodal WBC dataset for WBC classification; 2) We develop a high performing multimodal architecture which is also efficient and low in complexity at the same time.

Introduction

Automated leukocyte (WBC) classification is a cornerstone of modern medical diagnostics, providing essential data for identifying infections, inflammatory responses, and hematological disorders. Traditionally, these systems rely on a single imaging modality: either bright-field microscopy of stained blood smears, which offers rich structural detail but is labor-intensive, or fluorescence-based imaging, which is efficient but susceptible to photobleaching and provides less morphological detail.

While combining these modalities should theoretically improve diagnostic accuracy, it introduces a "fusion dilemma." Early Fusion (concatenating images at the input layer) is computationally efficient but often fails to capture the unique nuances of individual modalities. Conversely, Late Fusion (training separate networks for each modality and averaging their outputs) is highly accurate but multiplies the computational cost by the number of modalities used. This paper introduces a novel architecture, MM-MIMO, and a first-of-its-kind multimodal dataset to break this trade-off, achieving high-performance classification with a minimal computational footprint.

A First-of-its-Kind Multimodal WBC Dataset

A significant contribution of this research is the introduction of a unique multimodal WBC dataset. The dataset comprises 14,912 WBC samples, categorized into the five standard classes: Neutrophils, Lymphocytes, Eosinophils, Monocytes, and Basophils.

Each sample consists of four distinct images:

  • One Fluorescence image: Captured using Acridine Orange dye.
  • Three Bright-field images: Captured under different light sources (Color1, Color2, and Color3).

This dataset provides a new benchmark for multimodal cell analysis, allowing researchers to explore how different imaging techniques can complement one another in a clinical setting.

Methodology: MM-MIMO and Knowledge Distillation

The researchers proposed a two-step framework designed to maximize accuracy while maintaining the efficiency of a single backbone network.

1. Multi-modal Multi-input Multi-output (MM-MIMO)

Inspired by network redundancy concepts, the authors adapted the MIMO architecture to handle multimodal data. Instead of training four separate models, they trained independent modality-specific subnetworks within a single backbone. During the training phase, different WBC samples are fed into different "input slots." This forces the network to develop independent pathways and learn modality-specific features without relying on cross-modality correlations that might not exist or could lead to overfitting.

2. Knowledge Distillation (KD)

To further bridge the performance gap between a single backbone and a heavy ensemble, the framework employs a "Teacher-Student" approach. A high-complexity Late Fusion model—consisting of separate networks for each modality—acts as the teacher. This teacher provides "soft labels" to the MM-MIMO student model. Through this distillation process, the lightweight MM-MIMO model inherits the sophisticated feature-learning capabilities of the heavy ensemble without the associated computational overhead.

Experimental Results

The framework was evaluated using two different backbones: the efficient ShuffleNet V2 and the more robust ResNet34.

Accuracy and Performance

On the ShuffleNet V2 backbone, the proposed MM-MIMO + KD approach achieved a 95.99% F1-score. This performance not only significantly outperformed Early Fusion (95.43%) but also slightly surpassed the heavy Late Fusion ensemble (95.87%).

Computational Efficiency

The primary advantage of the MM-MIMO + KD approach is its efficiency. Compared to the Late Fusion ensemble, the proposed method reduced the parameter count and computational complexity by approximately 75%:

  • Late Fusion: ~5.03M parameters / 591.16M FLOPs.
  • MM-MIMO + KD: ~1.27M parameters / 172.19M FLOPs.

Modality Insights

Experimental analysis revealed that bright-field images (specifically the "Color3" light source) were more feature-rich than fluorescence images, which yielded the lowest individual performance. However, the highest accuracy was consistently achieved when all four modalities were utilized, proving the value of the multimodal approach.

Conclusion and Future Work

This work demonstrates that it is possible to achieve ensemble-level accuracy in WBC classification using a single, efficient network backbone. By combining the MM-MIMO architecture with Knowledge Distillation, the researchers have created a model that is both high-performing and suitable for deployment on standard clinical hardware.

Future research will focus on testing the framework on pathological samples, such as those from patients with leukemia or severe infections, where cell morphology is significantly distorted. Additionally, the authors suggest that this framework can serve as a guide for scaling multimodal architectures in other areas of medical imaging.

Summary updated

Abstract

Recently, a lot of automated white blood cells (WBC) or leukocyte classification techniques have been developed. However, all of these methods only utilize a single modality microscopic image i.e. either blood smear or fluorescence based, thus missing the potential of a better learning from multimodal images. In this work, we develop an efficient multimodal architecture based on a first of its kind multimodal WBC dataset for the task of WBC classification. Specifically, our proposed idea is developed in two steps - 1) First, we learn modality specific independent subnetworks inside a single network only; 2) We further enhance the learning capability of the independent subnetworks by distilling knowledge from high complexity independent teacher networks. With this, our proposed framework can achieve a high performance while maintaining low complexity for a multimodal dataset. Our unique contribution is two-fold - 1) We present a first of its kind multimodal WBC dataset for WBC classification; 2) We develop a high performing multimodal architecture which is also efficient and low in complexity at the same time.
BibTeX
@inproceedings{yang2022leukocyte,
  author = {Litao Yang and Deval Mehta and Dwarikanath Mahapatra and Zongyuan Ge},
  title = {{Leukocyte Classification using Multimodal Architecture Enhanced by Knowledge Distillation}},
  booktitle = {MOVI @ MICCAI},
  year = {2022},
  url = {https://openreview.net/forum?id=dchLJZGFxFL},
}
Figure