Abstract
Few-Shot Learning (FSL) aims to recognize new classes from limited examples, but performance often degrades in Cross-Domain Few-Shot Learning (CDFSL) where test domains differ significantly from training. While existing methods focus on adapting model parameters, the underlying optimization strategy typically remains static. We propose Task-Specific Preconditioned gradient descent (TSP), a framework that meta-learns domain-specific preconditioners and dynamically combines them based on task characteristics. By enforcing a positive definite constraint, TSP ensures stable and efficient convergence. Our approach achieves state-of-the-art performance on the Meta-Dataset benchmark while being significantly faster than previous adaptive optimization methods.
Introduction
The challenge of CDFSL lies in the drastic shifts in loss landscape geometry across diverse domains, such as transitioning from natural images (ImageNet) to satellite imagery or medical scans. Standard gradient descent (GD) uses a uniform learning rate and direction, which may be suboptimal for specialized domains. Task-Specific Preconditioned gradient descent (TSP) addresses this by adapting the optimizer itself to the specific geometry of each new task.
Methods
TSP replaces standard GD with Preconditioned Gradient Descent (PGD). The core idea is to warp the gradient using a "preconditioner" matrix to accelerate convergence. The framework relies on three components:
- Domain-Specific Preconditioners (DSPs): Unique matrices meta-learned for each training domain (e.g., ImageNet, Omniglot).
- Dataset Classifier: A lightweight network that estimates task-coefficients by comparing support images to known domains.
- Positive Definite (PD) Constraint: Preconditioners are parameterized as $P = M^T M + I$ to guarantee they remain positive definite, preventing gradient explosion and ensuring descent.
At inference, a task-specific preconditioner is formed by a linear combination of DSPs weighted by the predicted task-coefficients. This tailored optimizer then fine-tunes the model parameters for the specific few-shot task.
Key Insights
A critical observation is that no single optimizer is universally optimal across all tasks. As shown in Figure 3(a), the relative performance of Adadelta, GD, Adam, and RMSprop varies significantly between ImageNet and CIFAR-100 tasks. TSP effectively navigates this space by interpolating between learned optimization strategies.
Furthermore, the stability of adaptive optimization in unseen domains depends heavily on the Positive Definite (PD) constraint. Without it, accuracy on unseen domains drops precipitously, as the optimizer may fail to find a valid descent direction. Enforcing this constraint ensures the gradient always points in a "descent" direction, which is vital for generalization.
Results and Efficiency
Evaluated on the Meta-Dataset, TSP achieved state-of-the-art accuracy on 11 out of 13 datasets. Notably, it improved performance on unseen domains like Textures and Traffic Signs by 1.7% to 3.4%. Beyond accuracy, TSP is highly efficient; by avoiding expensive per-step operations like Singular Value Decomposition (SVD), it runs approximately 13 times faster during inference than previous adaptive methods like GAP.
Conclusion
TSP demonstrates that adapting the optimization strategy is as crucial as adapting model parameters in cross-domain scenarios. By meta-learning and blending positive-definite preconditioners, TSP provides a robust and efficient mechanism for CDFSL. Future work may explore non-linear combinations of domain knowledge and application to larger architectures like Vision Transformers.
Summary updated