ImagineFSL: Self-Supervised Pretraining Matters on Imagined Base Set for VLM-based Few-shot Learning
Abstract
Few-shot learning (FSL) remains a significant challenge for Vision-Language Models (VLMs) when data is extremely scarce. ImagineFSL addresses this by introducing a paradigm shift: treating synthetic images not merely as data augmentation, but as a standalone "imagined" pretraining set (iBase). By pretraining CLIP adapters on millions of LLM-curated synthetic images using a novel self-supervised method called HoM-DINO, ImagineFSL achieves new state-of-the-art results across 11 diverse recognition benchmarks.
Introduction
The emergence of Text-to-Image (T2I) models like Stable Diffusion has provided a source of virtually infinite labeled data. However, existing FSL methods often treat synthetic data as a secondary complement used only during final fine-tuning. ImagineFSL argues that synthetic data represents a distinct knowledge repository that should be leveraged for pretraining model adapters before encountering specific downstream tasks. This approach allows the model to build a robust foundation on an "imagined" base set before specializing on real-world few-shot examples.
Methodology
The ImagineFSL framework is built on three pillars: the iBase pipeline, HoM-DINO, and SyntAug.
The iBase Pipeline: To create a high-quality pretraining set, the authors use a "Chain-of-Thought" (CoT) approach with LLMs. GPT-4 analyzes key visual factors such as attributes, lighting, and viewpoint for a concept. Subsequently, a local Llama model generates thousands of diverse, contextually rich captions. These captions drive Stable Diffusion to generate the iBase (e.g., SyntIN1K), ensuring the synthetic data covers the "long tail" of visual appearances.
HoM-DINO (Higher-order Moment DINO): This is the core self-supervised learning algorithm. Unlike standard DINO which relies on a single [CLS] token, HoM-DINO models the distribution of all image patch tokens by calculating 1st, 2nd, and 3rd order moments (mean, variance, and skewness). This captures richer feature statistics essential for few-shot scenarios. Additionally, Masked Image Modeling (MIM) is incorporated to enhance local structure understanding.
SyntAug (Synthetic Augmentation): Instead of traditional geometric transforms, the authors use the T2I model to generate two visually diverse but semantically identical images from the same caption. This provides a much stronger contrastive signal for the self-supervised pretraining phase.
Experimental Results
ImagineFSL was evaluated on a rigorous suite of 11 benchmarks, including general object recognition (ImageNet, Caltech101), fine-grained classification (Oxford Pets, Flowers, Cars, Aircraft), and specialized domains (EuroSAT, DTD, SUN397, UCF101, and Food-101).
As shown in the results table, ImagineFSL consistently outperforms previous state-of-the-art methods like DISEF and DataDream. In the challenging 1-shot setting, it achieved an average accuracy of 75.8%, a significant jump over previous benchmarks. The method also demonstrates excellent scalability; upgrading from a ViT-B/16 to a ViT-L/14 backbone yielded nearly a 6% improvement. Furthermore, the fine-tuning stage is highly efficient, offering a two-fold speed improvement and using half the memory compared to other encoder-tuning methods.
Discussion and Conclusion
Key insights from this work highlight that self-supervised pretraining on synthetic data is superior to supervised pretraining, as it better bridges the domain gap between synthetic and real images. Capturing higher-order moments of patch tokens proves critical for robust representations in low-data regimes. While the initial pretraining on iBase requires significant computational resources (approx. 40 hours on 4x RTX 4090s), the resulting performance gains and downstream efficiency justify the investment. Future work will explore larger "imagined" sets and more advanced ways to align the synthetic and real feature spaces.
Summary updated