TL;DR
Distillation-based self-supervised learning (SSL) often suffers from "over-compression," where models focus too much on global semantics at the expense of local details. This paper introduces UDI (Unsqueezed Distillation), a framework that adds an extra class token and a multimodal target distribution to capture "meaningful nuisances," significantly improving performance in dense prediction tasks like object detection and segmentation.
What Problem Does This Paper Address?
Current state-of-the-art SSL methods, particularly those based on self-distillation like DINO and iBOT, are excellent at learning semantically meaningful representations for image classification. However, they tend to over-compress information. By using a single [CLS] token and a very sharp target distribution, these models effectively "squeeze" the representation to focus only on the most dominant object in an image.
While this is great for telling a dog from a cat, it discards the "nuisances"—the background details, textures, and local context—that are vital for downstream tasks like object detection and semantic segmentation. The authors argue that a truly rich representation should reflect the entire semantic composition of a natural image, not just its "main character."
Methods
The authors propose UDI (Unsqueezed Distillation-based SSL) to alleviate this bottleneck. The framework introduces three key innovations:
- Unsqueezing the Bottleneck: UDI adds an extra class token ($z_{cls+}$) to the Vision Transformer (ViT). While the standard $z_{cls}$ token focuses on global instance-level semantics, the new $z_{cls+}$ token is trained to produce multimodal predictions that capture a broader spectrum of information.
- Stratified Random Sampling (SRS) Module: To represent the image's diverse semantic components, UDI uses a grid-based sampling strategy. It selects patches across the image and processes them through a self-attention (SA) layer. This SA mechanism acts as a "soft-masked pooling," ensuring that patch-level representations are contextually aware and semantically aligned with the image level.
- Multimodal Target Distribution: Instead of a unimodal (sharp) target, UDI constructs a multimodal target by blending predictions from the global image level and various local patch levels. The $z_{cls+}$ token is then trained to match this richer, more complex distribution.
Datasets
The models were pre-trained on ImageNet-1K without labels. For evaluation, the authors utilized a wide range of standard benchmarks:
- Classification: ImageNet-1K (Linear probing, k-NN, low-shot, and fine-tuning).
- Dense Prediction: MS-COCO (Object Detection and Instance Segmentation) and ADE20K (Semantic Segmentation).
- Video/Transfer: DAVIS-2017 (Video Object Segmentation) and various smaller datasets like CIFAR-10/100, iNaturalist, and Flowers for transfer learning.
Key Insights and Ideas
- Meaningful Nuisances: The paper posits that "nuisances" (background info, local textures) are not just noise but essential components of a rich representation. By learning a multimodal distribution, UDI preserves these details.
- Forman Curvature: The authors link the "unsqueezing" of the bottleneck to improved graph curvature. By adding the extra token and its associated edges in the attention graph, they mitigate "over-squashing," a known issue in message-passing neural networks.
- Context-Aligned Constraints: Unlike previous methods that used discriminative patch-wise losses (which can be too harsh), UDI’s use of self-attention for patch representations provides a more "natural" semantic constraint that aligns better with how objects actually appear in images.
Key Results
- Superior Dense Prediction: UDI shows significant gains in dense tasks. On COCO object detection, it achieves 43.2 AP, outperforming iBOT (41.7) and DINO (40.8) under similar training conditions.
- Robust Low-Shot Learning: In the 1% label regime on ImageNet, UDI achieves 65.8% accuracy, a 3.9% improvement over iBOT, demonstrating that its richer representations are highly efficient.
- Competitive Classification: While focusing on richness, UDI remains top-tier in classification, reaching 77.6% linear top-1 accuracy on ImageNet-1K.
- Semantic Segmentation: On ADE20K, UDI achieves 43.7 mIoU (using UperNet), surpassing other multi-level SSL methods like SelfPatch.
Limitations and Future Work
The primary limitation noted is that UDI was designed and tested specifically for Vision Transformer (ViT) architectures. While ViTs are currently dominant in SSL, the authors suggest that future work could explore extending these "unsqueezing" principles to non-ViT backbones (like ConvNets). Additionally, the current study focuses on the ViT-S/16 model; evaluating the scalability of UDI with much larger models and longer training schedules remains an open direction.
Summary updated