Self-Supervised Learning as Discrete Communication

Kawtar Zaher, Ilyass Moummad, Olivier Buisson, Alexis Joly

Most modern self-supervised learning (SSL) architectures follow a familiar paradigm: take two different augmented "views" of the same image and train a neural network to align their embeddings in a continuous vector space. While highly effective, this continuous alignment approach often lacks interpretability. It offers little control over how information is structured across dimensions, frequently resulting in "entangled" features where a single dimension represents an inseparable mix of color, shape, and context.

In "Self-Supervised Learning as Discrete Communication," Kawtar Zaher and colleagues propose a fundamental shift. Instead of aligning continuous vectors, they frame visual self-supervised learning as a communication problem. In this setup, a "teacher" network transmits semantic information to a "student" network through a narrow, fixed-capacity binary channel.

The Core Problem: Continuous Entanglement

Traditional SSL methods, such as DINO or SimCLR, rely on global similarity. If two images are deemed similar, their entire embedding vectors are pushed closer together. However, visual scenes are inherently compositional. An image of a "person holding a fish" contains distinct semantic factors that continuous alignment does not explicitly separate. Furthermore, continuous spaces possess infinite precision, which can make optimization less stable or prone to collapsing into redundant dimensions.

The authors pose a critical question: Can we force a model to learn a structured "language" where each dimension—or bit—represents a distinct, reusable semantic attribute?

Methodology: The BITS Framework

To answer this, the authors introduce BITS (Binary Information Transmission for Self-Supervision), a framework that replaces continuous feature alignment with discrete binary agreement. The methodology relies on four key components:

  1. The Binary Channel: Similar to DINO, BITS utilizes a teacher-student architecture. However, the teacher’s output is passed through a sigmoid function and thresholded at 0.5 to generate a discrete binary code ($z \in \{0, 1\}^B$).
  2. Multi-Label Agreement: Instead of relying on a single prototype or cosine similarity loss, the student network is trained to predict every individual bit of the teacher's message using Binary Cross-Entropy (BCE).
  3. Coding-Rate Regularization: To prevent dimensional collapse—where the model transmits the same bit repeatedly—a coding-rate loss is applied. This encourages the bits to remain diverse across different images and uncorrelated with one another, thereby maximizing the information capacity of the binary channel.
  4. The "Reset" Trick (BITS-reset): In a powerful variant of the model, the projection head (the network component translating features into bits) is periodically reinitialized. This forces the backbone to learn robust, universal visual features capable of translating into any random binary mapping, preventing the model from overfitting to a single discretization scheme.

Key Insights and Semantic Reusability

Restricting the model's vocabulary to discrete bits proves to be a feature, not a bug. Because the communication channel is narrow, the model is forced into efficiency, yielding highly "factorized" representations where different bits capture distinct semantic concepts.

The learned bits form a compact, informative discrete language rather than random noise. For instance, the researchers identified specific bits that toggle based on the presence of a human in the scene, functioning independently of whether the primary object is a fish, a musical instrument, or another entity. Furthermore, the periodic "reset" acts as a strong regularizer, ensuring the backbone learns universal visual concepts rather than merely satisfying a static projection head.

Empirical Results

The BITS framework was evaluated across a wide array of datasets, including ImageNet-1K, COCO, and specialized domain-shift datasets like iNat2019 and PlantNet300k. The results demonstrate significant improvements over continuous baselines:

  • Superior Image Retrieval: BITS-reset achieved a remarkable 50.64 mAP on ImageNet retrieval, vastly outperforming DINO (35.68 mAP) and SimDINO (38.62 mAP).
  • Robustness to Domain Shift: Under severe domain shift on the iNat2019 naturalist dataset, BITS achieved 53.2% linear probing accuracy compared to DINO's 36.6%. This indicates that discrete agreement fosters features that are highly transferable and linearly separable.
  • High Efficiency: Even when the 256-bit code is subsampled down to just 32 bits, the model maintains a retrieval mAP of 40.35, still outperforming the full continuous embeddings of the DINO baseline.

Limitations and Future Directions

While BITS introduces a compelling new paradigm, the authors note a few limitations. The current study is limited to the ViT-B/16 architecture, and validating these findings at larger scales (ViT-L or ViT-G) remains a necessary next step. Additionally, integrating BITS with patch-level objectives—common in state-of-the-art models like DINOv2—presents an open challenge, as local patches lack clear multi-label semantic meanings. Finally, while binary codes are highly effective, future iterations may explore even richer discrete languages, such as learned vocabularies or symbolic tokens.

Summary updated

Abstract

Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input, offering limited control over how information is structured across representation dimensions. In this work, we frame visual self-supervised learning as a discrete communication process between a teacher and a student network, where semantic information is transmitted through a fixed-capacity binary channel. Rather than aligning continuous features, the student predicts multi-label binary messages produced by the teacher. Discrete agreement is enforced through an element-wise binary cross-entropy objective, while a coding-rate regularization term encourages effective utilization of the constrained channel, promoting structured representations. We further show that periodically reinitializing the projection head strengthens this effect by encouraging embeddings that remain predictive across multiple discrete encodings. Extensive experiments demonstrate consistent improvements over continuous agreement baselines on image classification, retrieval, and dense visual prediction tasks, as well as under domain shift through self-supervised adaptation. Beyond backbone representations, we analyze the learned binary codes and show that they form a compact and informative discrete language, capturing semantic factors reusable across classes.
BibTeX
@inproceedings{zaher2026self,
  author = {Kawtar Zaher and Ilyass Moummad and Olivier Buisson and Alexis Joly},
  title = {{Self-Supervised Learning as Discrete Communication}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=yVNMUeuZQd},
}
Figure