The dominant paradigm in self-supervised learning (SSL) has long been the "dual-branch" or Siamese network. Methods like SimCLR, MoCo, and BYOL work by taking two different augmented views of the same image and teaching a model to recognize that they represent the same underlying concept. While effective, this approach is computationally expensive: it requires twice the forward passes, massive batch sizes, and complex mechanisms like momentum encoders or memory banks to prevent the model from "collapsing" into a trivial solution.
A growing body of research is challenging this status quo by demonstrating that high-quality representations can be learned using single-branch architectures and single-view training. By treating SSL as a massive classification task or a structured reconstruction problem, these methods slash computational overhead while maintaining, or even exceeding, the performance of their dual-branch counterparts.
The Return of Instance Classification
The simplest way to avoid the complexity of dual-branch networks is to return to a classic idea: parametric classification. In this framing, every single image in a dataset is treated as its own unique category.
Yu Liu and colleagues demonstrated the power of this approach in Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning. By scaling a single-branch classifier to handle one million distinct classes, they showed that a model can learn robust features simply by trying to identify which specific image it is looking at. This removes the need for comparing pairs of images entirely, simplifying the training pipeline significantly.
Building on this, Yun-Hao Cao and Jianxin Wu refined the mathematics of instance discrimination in On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning. By introducing a "SqrtKL" self-distillation loss, they achieved a 71% reduction in training time compared to state-of-the-art dual-branch methods, proving that single-branch models are particularly effective for lightweight architectures and limited data scenarios.
Learning from a Single View
If we only have one view of an image, how does the model know what to learn? One answer is cluster discrimination. Instead of comparing an image to its augmented twin, the model compares an image to a set of globally maintained clusters.
Qi Qian and Rong Jin explored this in Unsupervised Visual Representation Learning by Online Constrained K-Means. Their approach uses a single augmented version of each image and assigns it to a cluster. The model learns by trying to predict these cluster assignments, focusing on global structure rather than local instance-level comparisons. This "single-view" strategy drastically reduces memory usage because the model doesn't need to store or process multiple versions of the same data point.
Another strategy is to use the model's own history as a reference point. In Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning, Chao Zhang and Hongyang Zhang proposed using an exponential moving average of the model's past predictions as a target. This creates a "memory" of what the image looks like, allowing the model to remain consistent over time without needing a second augmented branch in the current training step.
Masking as a Single-Branch Alternative
Masked modeling—popularized by BERT in NLP and MAE in vision—is inherently a single-branch approach. The model sees a partial version of the data and must predict the missing pieces.
Recent work has extended this to more complex domains:
- Audio: Aurian Quelennec and Slim Essid developed MATPAC, which combines masked latent prediction with patch classification. As detailed in Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning, this allows for high-quality audio features without the need for hand-crafted augmentations or dual-view encoders.
- Remote Sensing: Ankit Patnala applied a BERT-style masking strategy to satellite imagery in BERT Bi-modal self-supervised learning for crop classification using Sentinel-2 and Planetscope, showing that single-branch transformers are more scalable for massive unlabeled datasets than contrastive methods.
- Mixed Inputs: Kai Chen and Lanqing Hong introduced MixedAE in Mixed Autoencoder for Self-Supervised Visual Representation Learning. By mixing patches from different images and asking the model to recognize which patches belong together, they forced the model to learn object-aware representations within a single-branch autoencoder.
Why It Works: Efficiency and Stability
The move toward single-branch architectures is driven by a need for efficiency. Dual-branch methods are often "heavy," requiring specialized hardware and long training times to converge. By contrast, single-branch methods like those proposed by Ishan Rajendrakumar Dave in TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition focus on extracting multiple signals (like temporal invariance and distinctiveness) from the data itself rather than relying on expensive multi-modal inputs or redundant views.
Ultimately, these papers suggest that the "contrastive" signal provided by a second augmented view is not the only way to prevent representation collapse. Whether through massive instance classification, historical consistency, or masked reconstruction, single-branch architectures provide a leaner, faster, and more mathematically direct path to understanding the visual and auditory world.