Dynamic Weight Adjustment for Knowledge Distillation: Leveraging Vision Transformer for High-Accuracy Lung Cancer Detection and Real-Time Deployment

Saif Ur Rehman Khan, Muhammad Nabeel Asim, Sebastian J. Vollmer, Andreas DengelarXiv 2025 · DOI · arXiv

Introduction

Deep learning has revolutionized lung cancer diagnosis, yet a significant gap remains between model accuracy and clinical deployability. State-of-the-art architectures like Vision Transformers (ViTs) and deep Convolutional Neural Networks (CNNs) achieve high performance but are computationally prohibitive for real-time use in resource-constrained environments, such as mobile clinical devices. While Knowledge Distillation (KD) offers a path to compress these models by transferring knowledge from a large "teacher" to a smaller "student," traditional KD methods rely on fixed weights. This approach often fails in medical imaging, where noise and ambiguity require a more nuanced, adaptive learning strategy.

This paper introduces FuzzyDistillViT-MobileNet, a framework that utilizes fuzzy logic to dynamically adjust KD weights based on image uncertainty, enabling a lightweight MobileNet student to achieve performance comparable to heavy-duty architectures.

Methodology

Fuzzy Logic Dynamic Knowledge Distillation

The core innovation of this work is the fuzzy logic controller used during the distillation process. Traditional KD forces the student model to trust the teacher's soft labels equally across all images. However, medical images often contain regions of high uncertainty. The proposed controller takes the teacher’s confidence and the image’s inherent uncertainty as inputs. By applying overlapping fuzzy membership rules, it outputs a dynamic scaling weight for the KD loss. This ensures the student relies on the teacher when confidence is high and uncertainty is low, while reverting to ground-truth labels when the teacher's signal is ambiguous.

Model Selection and Image Fusion

To establish a robust teacher-student pair, the authors selected a Vision Transformer (ViT-B32) as the teacher for its global context extraction capabilities. A Genetic Algorithm (GA) was employed to screen 12 candidate CNNs, identifying MobileNet as the optimal student for its efficiency.

Before training, images undergo a pixel-level fusion process. Parallel enhancements—Gamma Correction for mid-tone adjustment and Histogram Equalization for contrast—are fused using a Discrete Wavelet Transform (wavedec2). This multi-scale approach preserves both spatial details and frequency features, providing a cleaner signal for the networks.

Experimental Setup and Datasets

The framework was validated across two distinct imaging modalities to ensure cross-domain robustness:

  1. LC25000 Dataset: 15,000 histopathological images across three classes (adenocarcinoma, squamous cell carcinoma, and benign tissue).
  2. IQ-OTH/NCCD Dataset: 1,097 CT-scan images categorized into benign, malignant, and normal cases.
Distribution of samples across the LC25000 and IQ-OTH/NCCD datasets used for training and validation.

Results and Discussion

FuzzyDistillViT-MobileNet achieved exceptional results, reaching 99.16% accuracy on the LC25000 dataset and 99.54% accuracy on the IQ-OTH/NCCD dataset. These metrics represent a significant improvement over standard MobileNet variants and other state-of-the-art (SOTA) approaches.

Comparison of the proposed method against recent literature in lung cancer detection.

As shown in the comparative analysis, the proposed method addresses the limitations of previous works—such as high computational intensity or reliance on static teacher models—by providing a lightweight yet highly accurate alternative. Ablation studies further confirmed that the fuzzy logic adjustment and the dual-enhancement image fusion were critical to these performance gains. To demonstrate real-world utility, the model was successfully deployed as an Android application, providing real-time diagnostic support on edge devices.

Conclusion

By replacing static distillation weights with a fuzzy logic-driven adaptive mechanism, this research successfully bridges the gap between high-accuracy ViTs and efficient mobile deployment. The FuzzyDistillViT-MobileNet model provides a robust, interpretable, and computationally efficient tool for lung cancer detection. Future work will focus on expanding the system to multi-modal data and further optimizing performance for devices with extremely limited hardware specifications.

Summary updated

BibTeX
@misc{khan2025dynamic,
  author = {Saif Ur Rehman Khan and Muhammad Nabeel Asim and Sebastian J. Vollmer and Andreas Dengel},
  title = {{Dynamic Weight Adjustment for Knowledge Distillation: Leveraging Vision Transformer for High-Accuracy Lung Cancer Detection and Real-Time Deployment}},
  year = {2025},
  eprint = {2510.20438},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20438},
}
Figure