EMPL: A novel Efficient Meta Prompt Learning Framework for Few-shot Unsupervised Domain Adaptation

Wanqi Yang, Haoran Wang, Lei Wang, Ge Song, Yang GaoarXiv 2024 · DOI · arXiv

Abstract

Few-shot Unsupervised Domain Adaptation (FS-UDA) is a challenging task that requires adapting models from a limited labeled source domain to an unlabeled target domain. Traditional methods often suffer from performance instability and high computational overhead due to iterative optimization. We present $E^2$MPL (Enduring and Efficient Meta Prompt Learning), a novel framework that utilizes a frozen CLIP backbone and a bilevel optimization strategy with closed-form solutions. $E^2$MPL significantly improves accuracy and stability while reducing adaptation time by approximately 70% compared to current state-of-the-art methods.

Introduction

Unsupervised Domain Adaptation (UDA) aims to generalize knowledge from a labeled source domain to an unlabeled target domain. In the few-shot setting (FS-UDA), the source domain is restricted to only a few labeled examples per category, exacerbating the domain gap. Existing FS-UDA approaches are often plagued by two issues: (1) high variance in performance across different tasks, and (2) low efficiency due to the need for iterative gradient-based updates during task adaptation. $E^2$MPL addresses these by introducing a meta-learning framework designed for both "endurance" (stability) and "efficiency."

Methodology

The $E^2$MPL framework keeps the pre-trained CLIP backbone frozen to preserve its rich general-purpose features and avoid overfitting on small few-shot datasets.

Dual-Prompt Design

The framework employs two types of prompts to capture different levels of information:

  • Domain-shared prompts: These learn general meta-knowledge across various tasks to help bridge the gap between source and target domains.
  • Task-specific prompts: These are generated by a lightweight prompt network that adaptively adjusts based on the specific images in the current task.

Bilevel Optimization with Closed-Form Solutions

The meta-learning process is structured as a bilevel optimization:

  • Inner Loop (Task Adaptation): Instead of iterative gradient descent, $E^2$MPL uses Ridge Regression and Cross-domain Projection. These have closed-form solutions, allowing the model to calculate optimal parameters in a single step, which ensures both speed and stability.
  • Outer Loop (Meta-Update): The model updates the prompt parameters and the prompt network by minimizing a meta-loss that includes classification loss, entropy minimization, and class discrimination across multiple meta-tasks.

Experimental Results

The authors evaluated $E^2$MPL on the DomainNet benchmark, testing 20 source-to-target domain pairs across 3,600 distinct tasks.

Accuracy and Stability

$E^2$MPL achieved a 15.4% improvement in accuracy for 1-shot tasks and an 8.7% improvement for 5-shot tasks over previous state-of-the-art methods. Beyond average accuracy, the model demonstrated remarkable stability across different domain shifts.

Accuracy boxplots for the skt->rel and rel->skt domain pairs. $E^2$MPL shows higher median accuracy and significantly tighter interquartile ranges (IQR) compared to baseline methods.

Efficiency

The use of closed-form solutions in the inner loop resulted in a 68.5% reduction in adaptation time for 1-shot tasks and a 74.1% reduction for 5-shot tasks. This efficiency allows for rapid deployment and testing across a vast number of tasks.

Conclusion

$E^2$MPL provides a robust and efficient framework for FS-UDA by combining meta-prompt learning with closed-form optimization. By achieving high accuracy with minimal variance and computational cost, it sets a new standard for enduring and efficient domain adaptation. Future research will focus on addressing class imbalance and extending the framework to more complex, specialized real-world domains.

Summary updated

BibTeX
@misc{yang2024empl,
  author = {Wanqi Yang and Haoran Wang and Lei Wang and Ge Song and Yang Gao},
  title = {{EMPL: A novel Efficient Meta Prompt Learning Framework for Few-shot Unsupervised Domain Adaptation}},
  year = {2024},
  eprint = {2407.04066},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.04066},
}
Figure