Abstract
Few-shot learning (FSL) remains a significant challenge in computer vision, primarily due to the tendency of models to rely on spurious correlations rather than intrinsic class features. This paper introduces the Time-step Few-shot (TiF) learner, a novel framework that leverages the internal feature hierarchy of Diffusion Models (DMs) to isolate nuanced class attributes. By parameterizing class-specific adapters based on diffusion time-steps, TiF effectively filters out coarse background noise and focuses on the fine-grained details necessary for accurate classification. Experimental results across specialized datasets, including aircraft identification and medical imaging, demonstrate that TiF significantly outperforms existing zero-shot and adapter-based baselines.
1. Introduction
Few-shot learning aims to recognize new categories from a limited number of training examples. Despite the success of foundation models like CLIP, these systems often struggle with "niche" categories—such as specific aircraft variants or individual person re-identification—where text prompts are insufficient. A recurring issue in FSL is the reliance on spurious correlations; for instance, a model might identify an "airplane" based on a blue sky background rather than the structural properties of the aircraft.
While standard adapters attempt to bridge this gap, they frequently latch onto visually prominent but irrelevant features. This paper proposes a different path: utilizing the generative logic of Diffusion Models to create a more robust discriminative learner.
2. The Time-step Bias in Diffusion Models
The core innovation of the TiF learner lies in its exploitation of the Time-step Bias. In a forward diffusion process, Gaussian noise is incrementally added to an image over $T$ time-steps. The authors observe a fundamental relationship between noise levels and feature scales:
- Early Time-steps (Small $t$): Nuanced, high-frequency, and fine-grained attributes are the first to be lost to noise.
- Later Time-steps (Large $t$): Coarse, low-frequency, and global structures (such as backgrounds and general shapes) persist much longer.
By focusing on the model's ability to reconstruct images at specific early time-steps, the TiF learner can effectively ignore the "easy" coarse features that lead to spurious correlations and instead prioritize the difficult, defining details of a category.
3. Methodology
3.1 Parameterization via LoRA
For each new class, the TiF learner trains a class-specific Low-Rank Adaptation (LoRA) module. This adapter is integrated into the attention blocks of a pre-trained Stable Diffusion U-Net. During training, the adapter is optimized to minimize the reconstruction error: given a noisy image $x_t$ and a generic prompt (e.g., "a photo of [V]"), the adapter must assist the model in recovering the original image $x_0$.
3.2 De-biasing via Weighted Reconstruction
Classification is performed by evaluating which class-specific adapter best reconstructs a given test image. However, rather than treating all time-steps equally, TiF employs a weighted average of reconstruction errors. The weights, denoted as $r_t$, are designed to peak at small values of $t$. This weighting scheme forces the classification decision to be based on the model's mastery of nuanced features.
Crucially, the method is hyper-parameter free. The authors derive an automated calculation for $r_t$ based on the pixel-level differences ($\delta^*$) between classes in the training set, ensuring the model adapts its focus based on the specific difficulty of the task.
4. Experimental Evaluation
4.1 Datasets
The TiF learner was evaluated on four diverse and challenging datasets:
- FGVCAircraft: 100 classes of aircraft requiring fine-grained distinction.
- ISIC2019: Medical images of skin lesions, testing performance in specialized domains.
- DukeMTMC-reID & VeRi-776: Person and vehicle re-identification tasks. These are particularly notable because they lack semantic class names, rendering standard CLIP-based zero-shot approaches ineffective.
4.2 Results
The TiF learner achieved state-of-the-art performance across all benchmarks:
- Accuracy Gains: On the 1-shot FGVCAircraft task, TiF improved accuracy by +24.2% over the zero-shot Diffusion Classifier baseline.
- Comparison to CLIP: TiF significantly outperformed OpenCLIP and Tip-Adapter. In re-identification tasks (DukeMTMC and VeRi), TiF surpassed baselines by up to 21.6%.
- Scalability: The method showed consistent performance gains as the number of shots increased from 1 to 16, demonstrating robust learning capabilities.
5. Discussion and Limitations
TiF represents a shift from purely discriminative FSL to a generative-informed approach. By forcing the model to reconstruct the class, it internalizes the actual features of the object rather than just finding a decision boundary.
However, limitations remain. If a spurious correlation is itself a fine-grained feature (e.g., a specific sensor artifact unique to one class's images), the time-step bias may not filter it out. Additionally, for very coarse-grained classification (e.g., "Dog" vs. "Car"), the current logic—which prioritizes early time-steps—might need inversion, as the defining features of those classes persist into later stages of diffusion.
6. Conclusion
The TiF learner demonstrates that the internal hierarchy of Diffusion Models provides a powerful inductive bias for few-shot learning. By parameterizing the learner according to diffusion time-steps, the framework successfully navigates the trade-off between coarse spuriousness and fine-grained accuracy, setting a new standard for FSL in complex, real-world domains.
Summary updated