MC-SSL: Towards Multi-Concept Self-Supervised Learning
Lists
Only you see this
Abstract
Self-supervised pre-training is the method of choice for natural language processing models and is rapidly gaining popularity in many vision tasks. Recently, self-supervised pre-training has shown to outperform supervised pre-training for many downstream vision applications, marking a milestone in the area. This superiority is attributed to the negative impact of incomplete labelling of the training images, which convey multiple concepts, but are annotated using a single dominant class label. Although Self-Supervised Learning (SSL), in principle, is free of this limitation, the choice of a pretext task facilitating SSL can perpetuate this shortcoming by driving the learning process towards a single concept output. This study aims to investigate the possibility of modelling all the concepts present in an image without using labels. In this respect the proposed Multi-Concept SSL (MC-SSL) framework is a step towards unsupervised learning which embraces all the diverse content in an image with the aim of explicitly modelling the information from all the concepts present in the image. MC-SSL involves two core design steps: group masked model learning (GMML) and learning of pseudo-concepts for data tokens using a momentum encoder (teacher-student) framework. An added benefit of MC-SSL is the ability to train data hungry transformers on small datasets with high accuracy without external data. Experimental results on multi-label and multi-class image classification downstream tasks demonstrate that MC-SSL not only surpasses existing SSL methods but also outperforms supervised transfer learning. The source code will be made publicly available for the community to train on bigger corpus.
BibTeX
@inproceedings{atito2023mc,
author = {Sara Atito and Muhammad Awais and Ammarah Farooq and Zhenhua Feng and Josef Kittler},
title = {{MC-SSL: Towards Multi-Concept Self-Supervised Learning}},
booktitle = {International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=uTshHIKOtan},
}