OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning

Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, Patrick Pérez

Abstract

Learning image representations without human supervision is an important and active research field. Several recent approaches have successfully leveraged the idea of making such a representation invariant under different types of perturbations, especially via contrastive-based instance discrimination training. Although effective visual representations should indeed exhibit such invariances, there are other important characteristics, such as encoding contextual reasoning skills, for which alternative reconstruction-based approaches might be better suited. With this in mind, we propose a teacher-student scheme to learn representations by training a convolutional net to reconstruct a bag-of-visual-words (BoW) representation of an image, given as input a perturbed version of that same image. Our strategy performs an online training of both the teacher network (whose role is to generate the BoW targets) and the student network (whose role is to learn representations), along with an online update of the visual-words vocabulary (used for the BoW targets). This idea effectively enables fully online BoW-guided unsupervised learning. Extensive experiments demonstrate the interest of our BoW-based strategy, which surpasses previous state-of-the-art methods (including contrastive-based ones) in several applications. For instance, in downstream tasks such Pascal object detection, Pascal classification and Places205 classification, our method improves over all prior unsupervised approaches, thus establishing new state-of-the-art results that are also significantly better even than those of supervised pre-training.

Introduction

Self-supervised learning (SSL) has traditionally relied on instance discrimination, where a model learns that different views of the same image should have similar representations. While this makes models invariant to certain transformations, it often fails to capture deep contextual reasoning—the understanding of how different parts of a scene relate to one another. Conversely, reconstruction-based methods like autoencoders often focus too much on low-level pixel noise rather than high-level semantic concepts.

OBoW (Online Bag-of-Visual-Words) addresses these limitations by training a network to reconstruct a "bag-of-visual-words" representation of an image from a small, distorted crop. By predicting abstract visual concepts rather than raw pixels, OBoW forces the model to develop sophisticated spatial dependencies. This approach has proven highly effective, outperforming popular contrastive learning methods on critical downstream tasks like object detection.

The OBoW Framework

OBoW utilizes a Teacher-Student architecture designed for fully online learning. Unlike previous BoW-based methods that required slow, offline vocabulary generation, OBoW updates its visual-word vocabulary and its teacher network dynamically during training.

Overview of the OBoW framework showing the teacher (T) generating BoW targets and the student (S) predicting them from perturbed inputs.

Teacher-Student Dynamics

As illustrated in Figure 1, the framework consists of two primary components:

  1. The Teacher ($T$): Processes a full image $\mathbf{x}$ to generate a target BoW representation $y_T(\mathbf{x})$. The teacher's weights are updated via a momentum-based moving average of the student's weights, ensuring a stable but evolving target.
  2. The Student ($S$): Receives a heavily perturbed, small crop $\tilde{\mathbf{x}}$ of the same image. Its goal is to predict the full BoW histogram generated by the teacher from the global view.

Online Vocabulary and Dynamic Prediction

A key innovation in OBoW is the use of a queue-based vocabulary $V$. This vocabulary consists of feature vectors sampled from the teacher during training. As the teacher improves, the "visual language" defined by the vocabulary evolves. To handle this changing target space, the student employs a Dynamic BoW Prediction Head.

The Dynamic BoW prediction head uses a sub-network $G(\cdot)$ to generate prediction weights from the current vocabulary $V$ on the fly.

As shown in Figure 3, the dynamic head $G(\cdot)$ takes the current vocabulary $V$ and produces a set of weights $G(V)$. These weights are then used in a dot-product operation with the student's feature vector $S(\tilde{\mathbf{x}})$ to produce the final prediction $y_S(\tilde{\mathbf{x}})$ via a softmax layer. This allows the student to adapt its output layer instantly to the evolving vocabulary, solving the "moving target" problem inherent in online SSL.

Experimental Evaluation

The models were primarily trained on ImageNet-1k and evaluated on several benchmarks to test generalization and contextual reasoning.

State-of-the-Art in Detection

On Pascal VOC object detection, OBoW achieved 82.9% $AP_{50}$, surpassing not only other self-supervised methods like MoCo v2 and SwAV but also beating supervised pre-training (81.3%). This suggests that the contextual pressure of reconstructing a global BoW from a local patch (sometimes less than 14% of the image area) is a powerful way to learn scene structure.

Efficiency and Few-Shot Learning

OBoW is remarkably efficient, reaching high accuracy with significantly fewer training epochs than competitors. It also excels when labeled data is scarce; on ImageNet with only 1% of labels, it achieved 82.9% top-5 accuracy, significantly higher than many contrastive counterparts.

Conclusion

OBoW introduces a robust paradigm for self-supervised learning by combining the strengths of reconstruction tasks with the efficiency of online teacher-student frameworks. By forcing a model to "hallucinate" global context from local patches using a dynamic bag-of-visual-words, it learns representations that excel at complex downstream tasks. Future research may explore combining this BoW reconstruction with contrastive losses or applying the framework to video data where temporal context is critical.

Summary updated

Abstract

Learning image representations without human supervision is an important and active research field. Several recent approaches have successfully leveraged the idea of making such a representation invariant under different types of perturbations, especially via contrastive-based instance discrimination training. Although effective visual representations should indeed exhibit such invariances, there are other important characteristics, such as encoding contextual reasoning skills, for which alternative reconstruction-based approaches might be better suited. With this in mind, we propose a teacher-student scheme to learn representations by training a convolutional net to reconstruct a bag-of-visual-words (BoW) representation of an image, given as input a perturbed version of that same image. Our strategy performs an online training of both the teacher network (whose role is to generate the BoW targets) and the student network (whose role is to learn representations), along with an online update of the visual-words vocabulary (used for the BoW targets). This idea effectively enables fully online BoW-guided unsupervised learning. Extensive experiments demonstrate the interest of our BoW-based strategy, which surpasses previous state-of-the-art methods (including contrastive-based ones) in several applications. For instance, in downstream tasks such Pascal object detection, Pascal classification and Places205 classification, our method improves over all prior unsupervised approaches, thus establishing new state-of-the-art results that are also significantly better even than those of supervised pre-training. We provide the implementation code at https://github.com/valeoai/obow.
BibTeX
@inproceedings{gidaris2021obow,
  author = {Spyros Gidaris and Andrei Bursuc and Gilles Puy and Nikos Komodakis and Matthieu Cord and Patrick Pérez},
  title = {{OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=CgCMRB27rY},
}
Figure