Poor Teaching: Explore and Question Knowledge Distillation under Distribution Shift

Songming Zhang, Ziyu Lyu, Xiaofeng ChenOpenReview 2024 · Submission to ICLR 2024 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Knowledge distillation techniques transfer knowledge from a complex or large learning model into a small model, and have made remarkable achievements in recent decades. However, few studies has investigated and explored the mechanism of the knowledge distillation against distribution shifts in real scenarios. In this paper, we reconsider the knowledge distillation paradigm under the shift situations, by reformulating the objectives of distillation with multiple domains. Under the novel paradigm, we propose a unified and systematic evaluation framework to benchmark knowledge distillation against two general distributional shifts including diversity and correlation shift. The evaluation benchmark covers more than 20 methods from algorithmic, data-driven, and optimization perspectives for five benchmark datasets. Extensive experiments are constructed and some constructive findings are summarized to explain when and how the existing knowledge distillation methods work against distribution shifts.
BibTeX
@misc{zhang2024poor,
  author = {Songming Zhang and Ziyu Lyu and Xiaofeng Chen},
  title = {{Poor Teaching: Explore and Question Knowledge Distillation under Distribution Shift}},
  year = {2024},
  url = {https://openreview.net/forum?id=2TOcJivjpt},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}

Loading related content...

Figure