Robust & Precise Knowledge Distillation-based Novel Context-Aware Predictor for Disease Detection in Brain and Gastrointestinal

Saif Ur Rehman Khan, Muhammad Nabeel Asim, Sebastian J. Vollmer, Andreas Dengel

Introduction

Medical image classification, particularly for detecting brain tumors and gastrointestinal diseases, remains a significant challenge in clinical diagnostics. The inherent variability in scan quality, presence of noise, and ambiguous anatomical features often hinder the performance of automated systems. To address these issues, researchers have turned to Knowledge Distillation (KD), a process where a complex "teacher" model transfers its expertise to a more efficient "student" model. However, traditional KD methods often struggle with data uncertainty, applying uniform smoothing that fails to account for varying image clarity.

The Problem: Static Scaling in Medical Imaging

In standard KD, the teacher's output probabilities are softened using a static "temperature" parameter. This approach treats all images identically, regardless of whether they are crystal-clear or highly distorted. Such uniform smoothing prevents the student model from learning how to handle ambiguous cases effectively, often leading to "inefficient variability" and poor training outcomes when faced with noisy data.

Overview of the Knowledge Distillation framework and common pitfalls addressed by the proposed model.

Proposed Methodology

This research introduces a novel framework that overcomes these limitations through two primary innovations:

1. Context-Aware Adaptive Temperature Scaling

Instead of a fixed temperature, the system employs a rule-based predictor to dynamically adjust the distillation temperature for each image. By analyzing image noise and teacher model confidence, the framework scales the temperature accordingly:

  • High Uncertainty: For noisy images where the teacher's confidence is low, the temperature is increased to provide softer labels, allowing the student to learn from the uncertainty.
  • High Confidence: For clear images where the teacher is certain, the temperature is decreased to provide harder labels, forcing the student to focus on reliable, high-precision information.

2. Ant Colony Optimization (ACO) for Model Selection

Selecting the optimal teacher-student architecture pair is typically a computationally expensive task. This framework utilizes Ant Colony Optimization (ACO) to navigate a pool of 16 pre-trained models. By simulating biological foraging behavior, the algorithm uses performance-based "pheromones" to efficiently converge on the best pairing: DenseNet201 as the teacher and ResNet152V2 as the student.

Experimental Setup and Datasets

The framework was rigorously evaluated on three benchmark datasets to ensure robustness across different medical domains:

  • Kaggle Multi MRI Dataset: 7,023 scans across four classes (Glioma, Meningioma, Pituitary, and No-tumor).
  • Figshare Multi MRI Dataset: 3,064 scans across three tumor classes.
  • GastroNet Dataset: 396 endoscopic images to test cross-domain generalization in gastrointestinal disease detection.
Sample MRI images and the distribution of training, validation, and test samples across the four primary classes.

Key Results and Discussion

The proposed Context-Aware Student Model (CASM) achieved state-of-the-art (SOTA) accuracy across all benchmarks, notably reaching 98.01% on the Kaggle MRI dataset. Beyond accuracy, the efficiency of the ACO selection process was remarkable, finding the optimal model pair in just 47 evaluations compared to the 1,000 required by traditional grid search methods.

Ablation studies further confirmed the model's robustness. Even when subjected to heavy artificial noise (Gaussian, Salt-and-Pepper), the performance remained stable, only dipping slightly to 95.19% in the most extreme noise scenarios. This demonstrates that dynamic temperature scaling effectively prevents the student from mimicking a teacher's overconfident errors on low-quality data.

Conclusion

By integrating context-aware scaling with biological optimization, this framework provides a robust solution for medical image classification. It successfully bridges the gap between high-performance complex models and efficient, deployable student models. Future work will focus on validating these results in real-world clinical environments and exploring integration into hospital monitoring systems for real-time diagnostic support.

Summary updated

BibTeX
@article{khan2026robust,
  author = {Saif Ur Rehman Khan and Muhammad Nabeel Asim and Sebastian J. Vollmer and Andreas Dengel},
  title = {{Robust \& Precise Knowledge Distillation-based Novel Context-Aware Predictor for Disease Detection in Brain and Gastrointestinal}},
  journal = {Biomedical Signal Processing and Control},
  year = {2026},
  doi = {10.1016/j.bspc.2025.108671},
}
Figure