Few-Shot Learner Parameterization by Diffusion Time-Steps

Zhongqi Yue, Pan Zhou, Richang Hong, Hanwang Zhang, Qianru Sun

Abstract

Few-shot learning (FSL) aims to recognize new categories from minimal examples but often fails due to spurious correlations—where models associate prominent background features with a class label instead of the actual object. This paper introduces the TiF (Time-step Few-shot) learner, which leverages the internal mechanics of Diffusion Models (DMs) to isolate subtle, class-defining attributes. By observing that fine-grained details are lost at low noise levels while backgrounds persist until high noise levels, we parameterize class-specific adapters to focus on these early-stage nuances. TiF significantly outperforms existing foundation model adapters in fine-grained classification and re-identification, demonstrating superior robustness to environmental noise.

Introduction

Training a model to recognize a new category from just a handful of examples is a fundamental challenge in computer vision. While foundation models like CLIP provide a strong starting point, they often struggle with fine-grained classification (e.g., distinguishing between similar aircraft variants). A primary reason for this failure is the model's tendency to rely on "spurious correlations"—prominent but irrelevant background features, such as a snowy runway or a clear blue sky, that happen to be present in the few available training images.

Existing adaptation methods for CLIP attempt to align image features with text, but they lack a structural mechanism to distinguish between the object of interest and its environment. This paper proposes a novel approach: using the time-steps of a Diffusion Model as a natural filter to separate nuanced class attributes from prominent spurious features.

The Diffusion Time-step Inductive Bias

The core insight of this work is that the forward diffusion process (adding noise to an image) acts as a hierarchical information filter. As noise increases over time-steps $t$, different types of visual information are lost at different rates:

  1. Early Time-steps (Low Noise): High-frequency, nuanced details—the very features that define a specific sub-class (like the shape of an aircraft window)—are the first to disappear.
  2. Late Time-steps (High Noise): Low-frequency, prominent features—such as the sky, ground, or general object silhouette—persist much longer.
The diffusion process acts as a natural filter, where fine-grained nuances are lost at early time-steps while backgrounds persist longer.

By analyzing the reconstruction error across these time-steps, we can identify a "sweet spot" where the model is forced to focus on class-defining nuances because the prominent background features have not yet been significantly degraded, or conversely, have already been accounted for by the base model's general knowledge.

Methodology: The TiF Learner

The TiF learner implements this bias through a parameterization strategy using Low-Rank Adaptation (LoRA). For each new class, a set of class-specific LoRA matrices ($\theta_c$) is trained to help a Diffusion Model reconstruct the original image from its noisy version.

To prevent the model from simply memorizing the training images, the low-rank nature of the adapters acts as a bottleneck, forcing the model to capture only the most essential attributes. During inference, the model calculates a weighted reconstruction error (TiF score) across multiple time-steps. A specialized weighting function, $r_t$, is applied to prioritize the errors at small time-steps where nuance loss is highest. This ensures the classification decision is based on the object's unique details rather than the background.

Experimental Results

The TiF learner was evaluated on challenging datasets including FGVCAircraft and DukeMTMC-reID. In fine-grained aircraft classification, TiF achieved a 13.7% improvement over state-of-the-art CLIP adapters like Tip-Adapter. Even more significantly, in person re-identification (reID) tasks, it saw gains of up to 21.6%.

TiF achieves state-of-the-art accuracy on FGVCAircraft and demonstrates superior focus on object-specific features compared to traditional adapters.

Qualitative analysis using attention maps (GradCAM) confirms that while traditional methods often "cheat" by looking at the sky or ground, TiF consistently focuses on the object's defining parts, such as wings or windows. This robustness to background noise is a direct result of the time-step parameterization, which effectively "de-biases" the learner.

Conclusion

The TiF learner represents a shift in few-shot learning by utilizing the temporal dynamics of Diffusion Models as a structural inductive bias. By focusing on the reconstruction of nuanced details lost at specific noise levels, the model avoids the pitfalls of spurious correlations. While diffusion-based inference is currently slower than single-pass encoders, the significant gains in accuracy and robustness suggest that time-step parameterization is a powerful tool for high-precision visual tasks.

Summary updated

Abstract

Even when using large multi-modal foundation models, few-shot learning is still challenging -- if there is no proper inductive bias, it is nearly impossible to keep the nuanced class attributes while removing the visually prominent attributes that spuriously correlate with class labels. To this end, we find an inductive bias that the time-steps of a Diffusion Model (DM) can isolate the nuanced class attributes, i.e., as the forward diffusion adds noise to an image at each time-step, nuanced attributes are usually lost at an earlier time-step than the spurious attributes that are visually prominent. Building on this, we propose Time-step Few-shot (TiF) learner. We train class-specific low-rank adapters for a text-conditioned DM to make up for the lost attributes, such that images can be accurately reconstructed from their noisy ones given a prompt. Hence, at a small time-step, the adapter and prompt are essentially a parameterization of only the nuanced class attributes. For a test image, we can use the parameterization to only extract the nuanced class attributes for classification. TiF learner significantly outperforms OpenCLIP and its adapters on a variety of fine-grained and customized few-shot learning tasks. Codes are in https://github.com/yue-zhongqi/tif.
BibTeX
@inproceedings{yue2024few,
  author = {Zhongqi Yue and Pan Zhou and Richang Hong and Hanwang Zhang and Qianru Sun},
  title = {{Few-Shot Learner Parameterization by Diffusion Time-Steps}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02195},
}
Figure