Reverse Knowledge Distillation: Training a Large Model Using a Small One for Retinal Image Matching on Limited Data

Sahar Almahfouz Nasser, Nihar Gupte, Amit SethiWACV 2024

TL;DR

This paper introduces "Reverse Knowledge Distillation" (RKD) to train large, powerful vision transformer models for retinal image matching, even with limited training data. By having a smaller, well-trained CNN model "teach" a larger transformer, they achieve better generalization and performance than traditional methods, demonstrating its effectiveness across retinal and facial landmark tasks.

What Problem Does This Paper Address?

Accurate retinal image matching (RIM) is crucial for monitoring eye health and disease progression. However, training state-of-the-art deep learning models, especially complex ones like vision transformers, requires vast amounts of labeled data. Such large-scale, precisely annotated datasets for RIM are scarce. This scarcity leads to a common problem: overfitting when training large models on limited data, hindering their performance and generalization. The paper aims to bridge this gap by enabling the effective training of powerful models on limited retinal image data.

Methods

The core innovation is Reverse Knowledge Distillation (RKD). Unlike traditional knowledge distillation where a large "teacher" model trains a smaller "student," RKD flips this: a smaller, pre-trained CNN model (like SuperRetina) acts as the teacher, guiding the training of a larger, more complex transformer-based model (the student).

The RKD framework involves:

  • Architectural Modifications: They adapt the SuperRetina architecture, which is a U-Net based semi-supervised method, by incorporating a transformer encoder (specifically, Swin UNETR).
  • RKD Loss Function: The training loss is a weighted combination of the standard detection and descriptor losses, plus two new distillation losses:
  • Detector RKD Loss: A dice loss between the keypoint heatmaps predicted by the student and teacher models.
  • Descriptor RKD Loss: A contrastive loss that matches descriptors from both student and teacher models.
  • Transfer Learning: To further address data scarcity for the transformer, they pre-train the Swin UNETR backbone on a larger, diverse dataset using tasks like image inpainting and angle prediction before fine-tuning it with RKD.
  • Dropout: A significant 50% dropout is applied to the transformer student model during RKD training, which surprisingly boosts performance and generalization.

Datasets

  • MeDAL-Retina Dataset: A newly curated dataset of 261 retinal images for training and validation. It combines images from existing public datasets (e-ophtha, retinal disease classification) and includes keypoint annotations at vessel intersections, crossovers, and bifurcations.
  • FIRE Dataset: A publicly available dataset of 129 retinal images used for testing and evaluation, comprising image pairs with varying degrees of overlap and deformation.
  • Wider Facial Landmarks in-the-wild (WFLW) Dataset: Used to demonstrate the generalizability of their RKD approach beyond retinal images, this dataset contains 10,000 faces with 98 landmarks per face.

Key Insights and Ideas

  • Flipping the Distillation Script: The most striking insight is that a smaller, well-established model can effectively "teach" a larger, more powerful model, especially when the larger model is prone to overfitting on limited data. This "reverse" direction of knowledge transfer proves beneficial.
  • RKD as a Regularizer: The RKD process, by forcing the larger student model to mimic the outputs of a simpler teacher, acts as a strong regularizer, preventing the student from overfitting to the limited training data and improving its generalization.
  • Transformer Generalization Boost: While transformers are known for their data hunger, RKD, combined with strategic dropout, allows a transformer student to surpass its CNN teacher and achieve excellent performance even on limited datasets.
  • Domain Generalization: The success of RKD on facial landmark detection suggests its applicability beyond medical imaging, highlighting its potential as a general technique for training large models on scarce data.

Key Results

  • Retinal Image Matching: Their RKD-trained Swin UNETR model (Ours-2) achieves state-of-the-art performance on the FIRE dataset, outperforming SuperRetina and other methods. Notably, it achieves 100% acceptable registrations and a high AUC across all categories.
  • Facial Landmark Detection: RKD-based SuperRetina (RKD-SR) demonstrates superior performance compared to both SuperRetina and Swin UNETR alone, achieving a 9.51% reduction in normalized mean error and showing robustness against outliers.
  • Ablation Studies: The paper shows that a blend of kernel sizes (1x1, 3x3, 5x5) in the CNN encoder improves performance, and while transfer learning helps, RKD is crucial for achieving top results with the transformer.

Limitations and Future Work

  • Limited Data for Transformer Pre-training: While they used a larger dataset for pre-training the Swin UNETR backbone, the initial pre-training dataset might still have distribution differences from the target retinal images, leading to suboptimal performance in some metrics before RKD.
  • RKD as a Generalization Tool: The paper suggests that RKD's ability to improve generalization is key, especially when dealing with limited data. Future work could explore RKD in other domains where data scarcity is a bottleneck for training large models.
  • Further Architectural Exploration: The authors mention architectural adjustments using CNN and transformer approaches to boost outcomes, implying room for further exploration in hybrid architectures.

Summary updated

BibTeX
@inproceedings{nasser2024reverse,
  author = {Sahar Almahfouz Nasser and Nihar Gupte and Amit Sethi},
  title = {{Reverse Knowledge Distillation: Training a Large Model Using a Small One for Retinal Image Matching on Limited Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure