TL;DR
This paper introduces DISEF, a novel framework for few-shot image classification that tackles the challenge of limited labeled data by generating diverse, in-domain synthetic images and efficiently fine-tuning Vision-Language Models (VLMs). DISEF achieves state-of-the-art results across ten benchmarks, demonstrating the power of combining advanced synthetic data generation with parameter-efficient fine-tuning.
What Problem Does This Paper Address?
Few-shot image classification aims to train models that can recognize new categories with very few labeled examples. This is crucial in domains where data collection is expensive or impractical. A major hurdle is learning generalizable representations from limited data. While using synthetic data from text-to-image models has shown promise, existing methods often generate out-of-domain images that can hurt performance, and they haven't fully explored efficient fine-tuning strategies for VLMs in this context. This paper addresses these limitations by proposing a method to generate diverse, in-domain synthetic data and a novel way to efficiently fine-tune VLMs for few-shot learning.
Methods
DISEF comprises two main components:
- Synthetic Augmentation Pipeline (SAP): This novel pipeline generates diverse, in-domain synthetic images. It leverages advanced captioning models (like LLaVA) to create detailed textual descriptions of real support images. These captions, along with the real images themselves, are used to guide a diffusion model (Stable Diffusion) to produce synthetic images that are semantically correct and visually similar to the real data. A filtering step using CLIP further ensures the generated images align with the target class.
- Efficient Fine-Tuning with LoRA: The authors propose using Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning of VLMs. Instead of updating all model parameters, LoRA injects small, low-rank update matrices into specific layers (query and value embeddings in self-attention layers) of both the vision and text encoders. This allows for effective adaptation to the few-shot task without modifying the original architecture or requiring extensive computational resources.
Datasets
The paper evaluates DISEF on ten commonly used benchmark datasets for few-shot image classification:
- Generic Object Classification: ImageNet, Caltech101
- Scene Understanding: SUN397
- Texture Classification: DTD
- Fine-Grained Classification: FGVC Aircraft, Oxford Pets, Stanford Cars, Food 101, Flowers 102
- Satellite Image Classification: EuroSAT
Key Insights and Ideas
- In-domain diversity is key: Simply generating synthetic images isn't enough; they must be semantically relevant and visually consistent with the real data to improve few-shot performance.
- Detailed captions matter: Leveraging rich, semantically detailed captions from models like LLaVA provides much stronger guidance for text-to-image generation than generic prompts.
- LoRA is effective for VLMs in few-shot settings: Parameter-efficient fine-tuning, specifically LoRA applied to both vision and text encoders, is a powerful and efficient way to adapt VLMs for few-shot tasks, outperforming other PEFT methods.
- Synthetic data complements real data: Combining real and synthetic data, generated with the proposed SAP, consistently improves performance across various datasets and fine-tuning methods.
Key Results
- State-of-the-art performance: DISEF consistently outperforms existing methods across ten benchmark datasets in the default (all classes) and base/new (unseen classes) scenarios.
- Significant improvements: In the base/new scenario, DISEF achieved a +1.55% average improvement in base accuracy and a +0.46% improvement in new accuracy compared to the best previous methods.
- Effective synthetic data generation: The SAP generates high-fidelity, realistic, and diverse in-domain images, as demonstrated by qualitative results and its positive impact on performance.
- Efficiency: LoRA-based fine-tuning significantly reduces the number of trainable parameters, making the adaptation process efficient.
Limitations and Future Work
- Pre-trained diffusion model limitations: The reliance on pre-trained diffusion models might limit the semantics of generated images for domains significantly different from the diffusion model's training data (e.g., medical images).
- Domain shift: If the domain shift between the pre-training data of the VLM (e.g., CLIP) and the few-shot task is too large, fine-tuning might not be sufficient.
- Data privacy: While not a technical limitation of the method itself, the paper notes that few-shot classification still relies on support sets, raising potential data privacy concerns.
- Bias: The potential for biases in generated data and their exacerbation in few-shot settings leading to unfair outcomes is mentioned as a broader societal impact.
Future work could explore adapting diffusion models to specific domains or investigating more robust VLM adaptation strategies for extreme domain shifts.