TL;DR
Universally Slimmable Self-Supervised Learning (US3L) addresses the challenge of deploying self-supervised models across diverse edge devices with varying hardware constraints. By establishing three core guidelines for temporal gradient consistency, US3L prevents the training collapse typical of naive slimmable self-supervised networks. With just a single training run and one set of weights, it delivers a single model capable of running at arbitrary channel widths while outperforming individually trained counterparts.
What Problem Does This Paper Address?
Self-supervised learning (SSL) has revolutionized computer vision by eliminating the need for massive labeled datasets. However, deploying these massive models to edge devices (like smartphones or embedded systems) is difficult due to strict memory and computational limits.
While model compression techniques like pruning and distillation help, they typically require training or fine-tuning a separate model for every target device constraint. "Slimmable networks" solve this in supervised learning by training a single network that can dynamically adjust its width (number of active channels) at runtime.
Unfortunately, when researchers try to naively swap supervised loss for self-supervised loss (like SimSiam) in a slimmable framework, the training process completely collapses or suffers severe performance drops. This paper investigates why this collapse happens and provides a robust framework to make slimmable self-supervised learning work seamlessly.
Methods
The authors introduce US3L (Universally Slimmable Self-Supervised Learning), which relies on three main innovations:
- Temporal Consistent Guidance (Loss Design): The authors analyze the gradients of common SSL losses and discover that Mean Squared Error (MSE)-based losses (like SimSiam) are highly unstable under rapid model updates and image augmentations. Conversely, Contrastive losses (like InfoNCE) are stabilized by relative distances to other samples. The authors outline three guidelines to maintain gradient stability, ultimately designing a framework that pairs an InfoNCE base loss with an auxiliary distillation MLP head and a momentum encoder target to keep sub-network training stable.
- Dynamic Sampling: Standard slimmable training uses the "sandwich rule," sampling multiple network widths at every single iteration, which is computationally expensive. US3L introduces a two-stage dynamic sampling strategy: it trains only the largest (base) network early on to establish a strong "teacher," and then gradually introduces progressively smaller sub-networks. This reduces the average number of sampled widths per iteration from 4 to 2.5.
- Group Regularization: In slimmable networks, early channels are active in every sub-network width, meaning they get updated far more frequently than later channels. This causes weights to concentrate heavily at the front, bottlenecking the base model's total capacity. Group regularization counteracts this by applying smaller weight decay coefficients to later channels, granting them more freedom to learn complex features.
Datasets
The authors evaluated US3L across several standard computer vision benchmarks:
- Pretraining & Linear Evaluation: CIFAR-10, CIFAR-100, and ImageNet.
- Downstream Transfer Tasks: Object detection and instance segmentation on Pascal VOC 07&12 and COCO2017; image recognition on Flowers, Pets, and Dtd.
Key Insights and Ideas
- The "Why" Behind the Collapse: The core "aha moment" is that MSE distillation forces sub-networks to directly and rigidly copy unstable, rapidly shifting targets across training iterations. InfoNCE prevents collapse because its denominator acts as a regularizer, anchoring the gradients relative to other data points in the batch.
- Target Alignment Matters: For stable distillation across varying network widths, all sub-networks must align to the exact same target (e.g., a slowly updating momentum teacher) rather than independent targets.
- Channel Freedom: Standard weight decay penalizes all channels equally, ignoring the architectural reality of slimmable networks. Loosening the regularization constraints on later channels significantly recovers the capacity of the full-sized model.
Key Results
US3L consistently outperforms individually trained models and standard slimmable baselines across CNNs (ResNet-18, ResNet-50, MobileNetv2) and Vision Transformers (ViT-Tiny, ViT-Small).
- ResNet-18 on CIFAR-100: At full width (1.0x), US3L achieves 69.0% linear accuracy (vs. BYOL's 66.8%). At the narrowest width (0.25x), US3L maintains 60.4% accuracy, outperforming BYOL by 6.1%—despite BYOL requiring a completely separate training run for that width.
- Vision Transformers: On CIFAR-10, US3L applied to a ViT-Small achieves 90.3% accuracy at full width and 85.5% at 0.25x width, significantly outperforming the MoCov3 baseline.
- Downstream Transfer: When transferred to Pascal VOC object detection, US3L provides up to a +18.5 $AP_{50}$ improvement over BYOL at the tightest 0.25x width constraint.
Limitations and Future Work
The primary limitation of this work is its focus on a single dimension of network slimmability: channel width. While width pruning is highly effective for hardware acceleration, real-world deployment scenarios also benefit from adaptations in network depth and kernel sizes. The authors note that future work will focus on expanding these temporal consistency guidelines to multi-dimensional slimmability and exploring integrations with Neural Architecture Search (NAS).
Summary updated