DETA: Denoised Task Adaptation for Few-Shot Learning

Ji Zhang, Lianli Gao, Xu Luo, Hengtao Shen, Jingkuan Song

Abstract

Test-time task adaptation in few-shot learning aims to adapt a pre-trained task-agnostic model for capturing task-specific knowledge of the test task, relying only on few-labeled support samples. Previous approaches generally focus on developing advanced algorithms to achieve the goal, while neglecting the inherent problems of the given support samples. In fact, with only a handful of samples available, the adverse effect of either the image noise (X-noise) or the label noise (Y-noise) from support samples can be severely amplified. To address this challenge, we propose DEnoised Task Adaptation (DETA), a unified image- and label-denoising framework orthogonal to existing task adaptation approaches. Without extra supervision, DETA filters out task-irrelevant, noisy representations by taking advantage of both global visual information and local region details of support samples. On the challenging Meta-Dataset, DETA consistently improves the performance of a broad spectrum of baseline methods applied on various pre-trained models. Notably, by tackling the overlooked image noise in Meta-Dataset, DETA establishes new state-of-the-art results.

Introduction

Few-Shot Learning (FSL) enables models to recognize new categories from very few examples (the "support set"). However, the scarcity of data makes models highly sensitive to imperfections in these samples. This research identifies two critical types of noise that plague FSL:

  1. Image Noise (X-noise): Cluttered backgrounds or occlusions that can lead a model to learn spurious correlations (e.g., associating "grass" with "dog" if the only dog image is in a park).
  2. Label Noise (Y-noise): Mislabeled samples which, in a low-data regime, can catastrophically degrade the model's understanding of a class.

While most FSL research focuses on adaptation algorithms assuming clean data, DETA recognizes that real-world support sets are inherently messy and that even small amounts of noise are severely amplified when data is scarce.

The DETA Framework

DETA is a plug-and-play framework designed to enhance existing FSL methods by denoising the support set during task adaptation. It combines local region analysis with global image context to identify and down-weight unreliable data.

The DETA framework: Contrastive Relevance Aggregation (CoRA) computes region weights to filter noise, while dual losses optimize local compactness and global dispersion.

Contrastive Relevance Aggregation (CoRA)

CoRA serves as a smart attention mechanism. It evaluates randomly cropped regions of support images, assigning higher weights to regions that show high in-class relevance and low out-of-class relevance. This effectively allows the model to "find" the object of interest and ignore background clutter.

Momentum Accumulator and Dual Losses

To stabilize the denoising process across training iterations, DETA employs a momentum-based accumulator for image and region weights. The framework is optimized using two complementary losses:

  • Local Compactness Loss ($L_l$): Pulls together the representations of high-weight (clean) regions from the same class.
  • Global Dispersion Loss ($L_g$): Pushes different class prototypes apart in the embedding space, specifically down-weighting the influence of images suspected of being mislabeled (Y-noise).

Experimental Results

The authors evaluated DETA on the Meta-Dataset (MD), currently the most rigorous benchmark in FSL, which aggregates ten diverse datasets including ImageNet, Omniglot, and CUB-Birds.

Qualitative analysis of region weights across diverse datasets. DETA successfully assigns lower weights to noisy regions and mislabeled samples.

Performance and Robustness

  • Standard Benchmarks: Even on "clean" datasets, DETA improved accuracy by 1.8% to 4.1% on average. This indicates that background noise is a pervasive but often ignored bottleneck in standard FSL benchmarks.
  • Label Noise Robustness: In scenarios with introduced label noise, DETA showed significant resilience. For instance, with 30% label noise, DETA improved the performance of a Swin Transformer baseline by 5.7%.
  • State-of-the-Art: By combining DETA with Task-Specific Adapters (TSA), the authors established a new state-of-the-art on Meta-Dataset, outperforming more complex meta-learning algorithms.

Discussion and Conclusion

DETA's primary strength lies in its orthogonality; it is not a standalone model but a refiner that can be integrated with various architectures (ResNets, ViTs) and adaptation strategies. By leveraging the synergy between local region details (to handle X-noise) and global image information (to handle Y-noise), DETA provides a unified solution for robust few-shot adaptation.

While the method introduces some computational overhead due to multiple crops and extra loss calculations, the performance gains in noisy, real-world scenarios justify the cost. Future work may explore extending these denoising principles to other modalities like Natural Language Processing.

Summary updated

Abstract

Test-time task adaptation in few-shot learning aims to adapt a pre-trained task-agnostic model for capturing taskspecific knowledge of the test task, rely only on few-labeled support samples. Previous approaches generally focus on developing advanced algorithms to achieve the goal, while neglecting the inherent problems of the given support samples. In fact, with only a handful of samples available, the adverse effect of either the image noise (a.k.a. X-noise) or the label noise (a.k.a. Y-noise) from support samples can be severely amplified. To address this challenge, in this work we propose DEnoised Task Adaptation (DETA), a first, unified image- and label-denoising framework orthogonal to existing task adaptation approaches. Without extra supervision, DETA filters out task-irrelevant, noisy representations by taking advantage of both global visual information and local region details of support samples. On the challenging Meta-Dataset, DETA consistently improves the performance of a broad spectrum of baseline methods applied on various pre-trained models. Notably, by tackling the overlooked image noise in Meta-Dataset, DETA establishes new state-of-the-art results. Code is released at https://github.com/nobody-1617/DETA.
BibTeX
@inproceedings{zhang2023deta,
  author = {Ji Zhang and Lianli Gao and Xu Luo and Hengtao Shen and Jingkuan Song},
  title = {{DETA: Denoised Task Adaptation for Few-Shot Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01060},
}
Figure