Masked Siamese ConvNets: Towards an Effective Masking Strategy for General-purpose Siamese Networks

Li Jing, Jiachen Zhu, Yann LeCunICLR 2023 · OpenReview

Abstract

Siamese Networks are a popular self-supervised learning framework that learns useful representation without human supervision by encouraging representations to be invariant to distortions. Existing methods heavily rely on hand-crafted augmentations, which are not easily adapted to new domains. To explore a general-purpose or domain-agnostic siamese network, we investigate using masking as augmentations in siamese networks. Recently, masking for siamese networks has only been shown useful with transformer architectures, e.g. MSN and data2vec. In this work, we identify the underlying problems of masking for siamese networks with arbitrary backbones, including ConvNets. We propose an effective and general-purpose masking strategy and demonstrate its effectiveness on various siamese network frameworks. Our method generally improves siamese networks' performances in the few-shot image classification, and object detection tasks.
BibTeX
@inproceedings{jing2023masked,
  author = {Li Jing and Jiachen Zhu and Yann LeCun},
  title = {{Masked Siamese ConvNets: Towards an Effective Masking Strategy for General-purpose Siamese Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=NnHz2rU0Hjp},
}

Loading related content...

Figure