Task-Specific Preconditioner for Cross-Domain Few-Shot Learning

Suhyun Kang, Jungwon Park, Wonseok Lee, Wonjong Rhee

Abstract

Few-Shot Learning (FSL) aims to recognize new classes from limited examples, but performance often degrades in Cross-Domain Few-Shot Learning (CDFSL) where test domains differ significantly from training. While existing methods focus on adapting model parameters, the underlying optimization strategy typically remains static. We propose Task-Specific Preconditioned gradient descent (TSP), a framework that meta-learns domain-specific preconditioners and dynamically combines them based on task characteristics. By enforcing a positive definite constraint, TSP ensures stable and efficient convergence. Our approach achieves state-of-the-art performance on the Meta-Dataset benchmark while being significantly faster than previous adaptive optimization methods.

Introduction

The challenge of CDFSL lies in the drastic shifts in loss landscape geometry across diverse domains, such as transitioning from natural images (ImageNet) to satellite imagery or medical scans. Standard gradient descent (GD) uses a uniform learning rate and direction, which may be suboptimal for specialized domains. Task-Specific Preconditioned gradient descent (TSP) addresses this by adapting the optimizer itself to the specific geometry of each new task.

Methods

TSP replaces standard GD with Preconditioned Gradient Descent (PGD). The core idea is to warp the gradient using a "preconditioner" matrix to accelerate convergence. The framework relies on three components:

  1. Domain-Specific Preconditioners (DSPs): Unique matrices meta-learned for each training domain (e.g., ImageNet, Omniglot).
  2. Dataset Classifier: A lightweight network that estimates task-coefficients by comparing support images to known domains.
  3. Positive Definite (PD) Constraint: Preconditioners are parameterized as $P = M^T M + I$ to guarantee they remain positive definite, preventing gradient explosion and ensuring descent.
Conceptual overview of Task-Specific Preconditioner (TSP) blending domain-specific preconditioners ($P_1, P_2, P_3$) into a target task preconditioner ($P_T$).

At inference, a task-specific preconditioner is formed by a linear combination of DSPs weighted by the predicted task-coefficients. This tailored optimizer then fine-tunes the model parameters for the specific few-shot task.

Key Insights

A critical observation is that no single optimizer is universally optimal across all tasks. As shown in Figure 3(a), the relative performance of Adadelta, GD, Adam, and RMSprop varies significantly between ImageNet and CIFAR-100 tasks. TSP effectively navigates this space by interpolating between learned optimization strategies.

Furthermore, the stability of adaptive optimization in unseen domains depends heavily on the Positive Definite (PD) constraint. Without it, accuracy on unseen domains drops precipitously, as the optimizer may fail to find a valid descent direction. Enforcing this constraint ensures the gradient always points in a "descent" direction, which is vital for generalization.

Empirical motivation for TSP: (a) Performance of standard optimizers varies by task; (b) The Positive Definite (PD) constraint is essential for generalization to unseen domains.

Results and Efficiency

Evaluated on the Meta-Dataset, TSP achieved state-of-the-art accuracy on 11 out of 13 datasets. Notably, it improved performance on unseen domains like Textures and Traffic Signs by 1.7% to 3.4%. Beyond accuracy, TSP is highly efficient; by avoiding expensive per-step operations like Singular Value Decomposition (SVD), it runs approximately 13 times faster during inference than previous adaptive methods like GAP.

Conclusion

TSP demonstrates that adapting the optimization strategy is as crucial as adapting model parameters in cross-domain scenarios. By meta-learning and blending positive-definite preconditioners, TSP provides a robust and efficient mechanism for CDFSL. Future work may explore non-linear combinations of domain knowledge and application to larger architectures like Vision Transformers.

Summary updated

Abstract

Cross-Domain Few-Shot Learning~(CDFSL) methods typically parameterize models with task-agnostic and task-specific parameters. To adapt task-specific parameters, recent approaches have utilized fixed optimization strategies, despite their potential sub-optimality across varying domains or target tasks. To address this issue, we propose a novel adaptation mechanism called Task-Specific Preconditioned gradient descent~(TSP). Our method first meta-learns Domain-Specific Preconditioners~(DSPs) that capture the characteristics of each meta-training domain, which are then linearly combined using task-coefficients to form the Task-Specific Preconditioner. The preconditioner is applied to gradient descent, making the optimization adaptive to the target task. We constrain our preconditioners to be positive definite, guiding the preconditioned gradient toward the direction of steepest descent. Empirical evaluations on the Meta-Dataset show that TSP achieves state-of-the-art performance across diverse experimental scenarios.
BibTeX
@inproceedings{kang2025task,
  author = {Suhyun Kang and Jungwon Park and Wonseok Lee and Wonjong Rhee},
  title = {{Task-Specific Preconditioner for Cross-Domain Few-Shot Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i17.33953},
}

Loading related content...

Figure