Rethinking Intermediate Layers Design in Knowledge Distillation for Kidney and Liver Tumor Segmentation

Vandan Gorade, Sparsh Mittal, Debesh Jha, Ulas Bagci

Medical image segmentation—the process of digitally outlining organs or tumors in scans—is a high-stakes task. While deep learning models have become incredibly good at this, the best ones are often massive, "heavy" architectures that are difficult to deploy on standard hospital hardware. Knowledge Distillation (KD) is the go-to solution: you train a small "student" model to mimic a large "teacher" model.

However, a new paper from researchers at Northwestern University and IIT Roorkee suggests we’ve been doing KD for medical imaging slightly wrong. By rethinking how information flows from teacher to student, they’ve achieved massive performance gains in kidney and liver tumor segmentation.

TL;DR

The authors introduce Hierarchical Layer-selective Feedback Distillation (HLFD), a method that strategically transfers knowledge from a teacher's deep and middle layers back to a student's earlier layers. This "feedback" approach helps compact student models ignore noise and focus on tumor-specific details, outperforming standard student models by over 10% in Dice score.

What Problem Does This Paper Address?

In standard Knowledge Distillation, the student usually tries to match the teacher's final output or mimic specific intermediate layers one-to-one. The authors identify two major flaws in this approach for medical imaging:

  1. Training Bias: When supervision only comes from the final layer, the signal weakens as it backpropagates. This leads to "training bias" in the shallower (earlier) layers of the student, which never quite learn to extract high-quality features.
  2. The "What" and "Where" Gap: Existing methods often lack a strategy for which teacher layers should talk to which student layers. Simply mimicking a corresponding layer might miss global context that a deeper teacher layer has already captured.

In medical tasks like tumor segmentation, tumors are often irregular, small, and surrounded by "noisy" anatomical structures. If the student's early layers aren't guided correctly, they fail to suppress this irrelevant information.

Methods

The core innovation is HLFD (Hierarchical Layer-selective Feedback Distillation). Instead of a simple top-down or layer-to-layer transfer, it uses a "feedback" mechanism across two levels:

1. Feature-level LFD (FLFD)

  • Unified Feature-level Distillation (UFD): It takes a combination of the teacher’s middle layers and distills that "unified" knowledge into the student’s early layers.
  • Individual Feature-level Distillation (IFD): It takes the teacher’s final representation and distills it into the student’s middle layers.

2. Pixel-level LFD (PLFD)

The method doesn't just look at abstract features; it looks at the actual predicted segmentation maps (pixels) at different stages of the decoder.

  • Unified Pixel-level Distillation (UPD): Teacher’s middle-layer predictive maps guide the student’s early maps.
  • Individual Pixel-level Distillation (IPD): Teacher’s final predictive map guides the student’s intermediate maps.

By using these "feedback" paths, the student's early layers are forced to learn representations that are already "aware" of what the deeper layers are looking for.

Datasets

The researchers evaluated their method on two major benchmarks:

  • KiTS (Kidney Tumor Segmentation): 210 abdominal CT scans.
  • LiTS (Liver Tumor Segmentation): 201 CT scans.

They used a UNet++ (36.1M parameters) as the teacher and a much smaller ResNet18 (11.6M parameters) as the student.

Key Insights and Ideas

  • Feedback is Better than Mirroring: The most significant "aha moment" is that an early student layer benefits more from seeing a teacher's processed middle-layer knowledge than from just trying to copy the teacher's own early (and often noisy) layers.
  • Noise Suppression: Qualitative GradCAM visualizations (heatmaps of what the AI is looking at) showed that HLFD allows the student to ignore irrelevant organs and "sharpen" its focus on the tumor much earlier in the network's architecture.
  • Multi-Task Synergy: Combining feature-level and pixel-level distillation in a multi-task loss function ($L_H$) ensures the model understands both the "what" (features) and the "where" (precise pixel boundaries).

Key Results

The quantitative improvements were substantial, especially considering the student model remains lightweight:

  • Kidney Segmentation (KiTS): The HLFD student achieved a Dice Similarity Coefficient (DSC) of 52.18%, compared to just 41.30% for a student trained without KD—a 10.8% absolute improvement.
  • Liver Segmentation (LiTS): HLFD reached 48.75% DSC, outperforming other state-of-the-art KD methods like SKD and DKD.
  • Volume Accuracy: On the Relative Volume Difference (RVD) metric, HLFD showed a much closer match to ground truth volumes than baselines, which is critical for clinical staging of tumors.
  • Qualitative Superiority: Visualizations showed that while other KD methods often produced "noisy" segmentations at intermediate layers, HLFD's intermediate layers were already remarkably focused on the tumor ROI.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance depends on balancing the weights ($\beta$ and $\lambda$) of the feature and pixel losses. While the authors provided a sensitivity analysis, finding the "sweet spot" for new datasets might require tuning.
  • Architectural Scope: The study focused on UNet++ and ResNet18. While these are industry standards, testing the feedback mechanism on Transformer-based models (like Swin-UNet) would be a logical next step.
  • Inference Speed: While the student is "compact," the paper doesn't explicitly detail the frames-per-second (FPS) gains on specific edge hardware, which would be valuable for real-time surgical applications.

Summary updated

Abstract

Knowledge distillation (KD) has demonstrated remarkable success across various domains, but its application to medical imaging tasks, such as kidney and liver tumor segmentation, has encountered challenges. Many existing KD methods are not specifically tailored for these tasks. Moreover, prevalent KD methods often lack a careful consideration of ‘what’ and ‘from where’ to distill knowledge from the teacher to the student. This oversight may lead to issues like the accumulation of training bias within shallower student layers, potentially compromising the effectiveness of KD. To address these challenges, we propose Hierarchical Layer-selective Feedback Distillation (HLFD). HLFD strategically distills knowledge from a combination of middle layers to earlier layers and transfers final layer knowledge to intermediate layers at both the feature and pixel levels. This design allows the model to learn higher-quality representations from earlier layers, resulting in a robust and compact student model. Extensive quantitative evaluations reveal that HLFD outperforms existing methods by a significant margin. For example, in the kidney segmentation task, HLFD surpasses the student model (without KD) by over 10%, significantly improving its focus on tumor-specific features. From a qualitative standpoint, the student model trained using HLFD excels at suppressing irrelevant information and can focus sharply on tumor-specific details, which opens a new pathway for more efficient and accurate diagnostic tools.
BibTeX
@inproceedings{gorade2024rethinking,
  author = {Vandan Gorade and Sparsh Mittal and Debesh Jha and Ulas Bagci},
  title = {{Rethinking Intermediate Layers Design in Knowledge Distillation for Kidney and Liver Tumor Segmentation}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=6I7Tf6OV6E},
}
Figure