Distilling Distributional Uncertainty from a Gaussian Process
Lists
Only you see this
Abstract
A Neural Network (NN) may exhibit overconfidence about wrong hypotheses, especially for Out-Of-Domain (OOD) inputs. A Gaussian process (GP) instead has an explainable distributional uncertainty behaviour, by predicting hypotheses with greater uncertainty for query inputs further from the training data. Previous work has shown that a NN can learn to emulate the behaviour of a GP on in-domain data. This paper expands upon this, by proposing to train a NN student to emulate the GP teacher’s distributional uncertainty behaviour on OOD data. This avoids the computational cost of using a GP at run-time, while improving the OOD confidence calibration of a NN. More accurate confidence calibration may better inform how the system should feedback to the user. Experiments on the SEP-28k-E stutter detection dataset suggest that distillation of such knowledge is feasible between these models.
BibTeX
@inproceedings{wong2024distilling,
author = {Jeremy H. M. Wong and Nancy F. Chen},
title = {{Distilling Distributional Uncertainty from a Gaussian Process}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2024},
doi = {10.1109/icassp48485.2024.10448172},
}Loading related content...