Abstract
Recently, a lot of automated white blood cells (WBC) or leukocyte classification techniques have been developed. However, all of these methods only utilize a single modality microscopic image i.e. either blood smear or fluorescence based, thus missing the potential of a better learning from multimodal images. In this work, we develop an efficient multimodal architecture based on a first of its kind multimodal WBC dataset for the task of WBC classification. Specifically, our proposed idea is developed in two steps - 1) First, we learn modality specific independent subnetworks inside a single network only; 2) We further enhance the learning capability of the independent subnetworks by distilling knowledge from high complexity independent teacher networks. With this, our proposed framework can achieve a high performance while maintaining low complexity for a multimodal dataset. Our unique contribution is two-fold - 1) We present a first of its kind multimodal WBC dataset for WBC classification; 2) We develop a high performing multimodal architecture which is also efficient and low in complexity at the same time.
Introduction
Automated leukocyte (WBC) classification is a cornerstone of modern medical diagnostics, providing essential data for identifying infections, inflammatory responses, and hematological disorders. Traditionally, these systems rely on a single imaging modality: either bright-field microscopy of stained blood smears, which offers rich structural detail but is labor-intensive, or fluorescence-based imaging, which is efficient but susceptible to photobleaching and provides less morphological detail.
While combining these modalities should theoretically improve diagnostic accuracy, it introduces a "fusion dilemma." Early Fusion (concatenating images at the input layer) is computationally efficient but often fails to capture the unique nuances of individual modalities. Conversely, Late Fusion (training separate networks for each modality and averaging their outputs) is highly accurate but multiplies the computational cost by the number of modalities used. This paper introduces a novel architecture, MM-MIMO, and a first-of-its-kind multimodal dataset to break this trade-off, achieving high-performance classification with a minimal computational footprint.
A First-of-its-Kind Multimodal WBC Dataset
A significant contribution of this research is the introduction of a unique multimodal WBC dataset. The dataset comprises 14,912 WBC samples, categorized into the five standard classes: Neutrophils, Lymphocytes, Eosinophils, Monocytes, and Basophils.
Each sample consists of four distinct images:
- One Fluorescence image: Captured using Acridine Orange dye.
- Three Bright-field images: Captured under different light sources (Color1, Color2, and Color3).
This dataset provides a new benchmark for multimodal cell analysis, allowing researchers to explore how different imaging techniques can complement one another in a clinical setting.
Methodology: MM-MIMO and Knowledge Distillation
The researchers proposed a two-step framework designed to maximize accuracy while maintaining the efficiency of a single backbone network.
1. Multi-modal Multi-input Multi-output (MM-MIMO)
Inspired by network redundancy concepts, the authors adapted the MIMO architecture to handle multimodal data. Instead of training four separate models, they trained independent modality-specific subnetworks within a single backbone. During the training phase, different WBC samples are fed into different "input slots." This forces the network to develop independent pathways and learn modality-specific features without relying on cross-modality correlations that might not exist or could lead to overfitting.
2. Knowledge Distillation (KD)
To further bridge the performance gap between a single backbone and a heavy ensemble, the framework employs a "Teacher-Student" approach. A high-complexity Late Fusion model—consisting of separate networks for each modality—acts as the teacher. This teacher provides "soft labels" to the MM-MIMO student model. Through this distillation process, the lightweight MM-MIMO model inherits the sophisticated feature-learning capabilities of the heavy ensemble without the associated computational overhead.
Experimental Results
The framework was evaluated using two different backbones: the efficient ShuffleNet V2 and the more robust ResNet34.
Accuracy and Performance
On the ShuffleNet V2 backbone, the proposed MM-MIMO + KD approach achieved a 95.99% F1-score. This performance not only significantly outperformed Early Fusion (95.43%) but also slightly surpassed the heavy Late Fusion ensemble (95.87%).
Computational Efficiency
The primary advantage of the MM-MIMO + KD approach is its efficiency. Compared to the Late Fusion ensemble, the proposed method reduced the parameter count and computational complexity by approximately 75%:
- Late Fusion: ~5.03M parameters / 591.16M FLOPs.
- MM-MIMO + KD: ~1.27M parameters / 172.19M FLOPs.
Modality Insights
Experimental analysis revealed that bright-field images (specifically the "Color3" light source) were more feature-rich than fluorescence images, which yielded the lowest individual performance. However, the highest accuracy was consistently achieved when all four modalities were utilized, proving the value of the multimodal approach.
Conclusion and Future Work
This work demonstrates that it is possible to achieve ensemble-level accuracy in WBC classification using a single, efficient network backbone. By combining the MM-MIMO architecture with Knowledge Distillation, the researchers have created a model that is both high-performing and suitable for deployment on standard clinical hardware.
Future research will focus on testing the framework on pathological samples, such as those from patients with leukemia or severe infections, where cell morphology is significantly distorted. Additionally, the authors suggest that this framework can serve as a guide for scaling multimodal architectures in other areas of medical imaging.
Summary updated