Dynamic Self-Supervised Teacher-Student Network Learning
Lists
Only you see this
Abstract
Lifelong learning (LLL) represents the ability of an artificial intelligence system to learn successively a sequence of different databases. In this paper we introduce the Dynamic Self-Supervised Teacher-Student Network (D-TS), representing a more general LLL framework, where the Teacher is implemented as a dynamically expanding mixture model which automatically increases its capacity to deal with a growing number of tasks. We propose the Knowledge Discrepancy Score (KDS) criterion for measuring the relevance of the incoming information characterizing a new task when compared to the existing knowledge accumulated by the Teacher module from its previous training. The KDS ensures a light Teacher architecture while also enabling to reuse the learned knowledge whenever appropriate, accelerating the learning of given tasks. The Student module is implemented as a lightweight probabilistic generative model. We introduce a novel self-supervised learning procedure for the Student that allows to capture cross-domain latent representations from the entire knowledge accumulated by the Teacher as well as from novel data. We perform several experiments which show that D-TS can achieve the state of the art results in LLL while requiring fewer parameters than other methods.
BibTeX
@article{ye2023dynamic,
author = {Fei Ye and Adrian G. Bors},
title = {{Dynamic Self-Supervised Teacher-Student Network Learning}},
journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
year = {2023},
doi = {10.1109/tpami.2022.3220928},
}Loading related content...