Boosting Prompt-Based Few-Shot Learners Through Out-of-Domain Knowledge Distillation
Lists
Only you see this
Abstract
Prompt-based learning improves the performance of Pre-trained Language Models (PLMs) over few-shot learning and is suitable for low-resourced scenarios. However, it is challenging to deploy large PLMs online. Knowledge Distillation (KD) can compress large PLMs into small ones; yet, few-shot KD for prompt-tuned PLMs is challenging due to the lack of training data and the capacity gap between teacher and student models. We propose Boost-Distiller, the first few-shot KD algorithm for prompt-tuned PLMs with the help of the out-of-domain data. Apart from distilling the model logits, Boost-Distiller specifically considers heuristically-generated fake logits that improve the generalization abilities of student models. We further leverage the cross-domain model logits, weighted with domain expertise scores that measure the transferablity of out-of-domain instances. Experiments over various datasets show Boost-Distiller consistently outperforms baselines by a large margin.
BibTeX
@inproceedings{chen2023boosting,
author = {Xiaoqing Chen and Chengyu Wang and Junwei Dong and Minghui Qiu and Liang Feng and Jun Huang},
title = {{Boosting Prompt-Based Few-Shot Learners Through Out-of-Domain Knowledge Distillation}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2023},
url = {https://openreview.net/forum?id=2TCmETiHm12},
}Loading related content...