Few-Shot Learning via Repurposing Ensemble of Black-Box Models

Minh Hoang, Trong Nghia HoangAAAI 2024 · DOI

Abstract

This paper introduces a framework for "black-box repurposing," allowing researchers to solve new tasks with very little data by leveraging an ensemble of existing, frozen AI models. By learning to intelligently transform inputs to "prompt" these black-box APIs for useful features, the authors achieve high-performance few-shot learning without ever needing access to the models' internal weights or original training data.

Introduction

In the current deep learning landscape, leveraging pre-trained models typically involves transfer learning or meta-learning. However, these techniques generally require "white-box" access to internal parameters (weights) or the massive datasets used for initial training. In contrast, many state-of-the-art models are deployed as "Black-Box" APIs (Machine Learning-as-a-Service, or MLaaS), where internal architectures are hidden. This research addresses the challenge of using these restricted, heterogeneous models to solve new "target" tasks in few-shot settings, where only a handful of labeled examples are available.

Methodology: Model Fusion via Repurposing (MFR)

The proposed Model Fusion via Repurposing (MFR) framework avoids modifying black-box models directly. Instead, it focuses on modifying the input sent to them to elicit useful feature responses.

The MFR architecture utilizes adaptation flows and latent factorization to integrate features from multiple black-box models.

The framework consists of four primary components:

  1. Input Augmentation via Normalizing Flows: Invertible transformations (e.g., Real NVP) warp target task inputs into a format that triggers useful responses from the black-box models.
  2. Latent Factorization: A Variational Autoencoder (VAE) approach decomposes augmented inputs into a common factor (shared knowledge) and a specific factor (task-unique knowledge).
  3. Ensemble Integration: Outputs from multiple models are combined. This allows models trained on disparate domains (e.g., digits vs. animals) to contribute to a rich feature set for the new task.
  4. Random Gradient Estimation: To overcome the non-differentiable nature of black-box APIs, the authors use a finite difference trick, perturbing inputs to estimate the gradients required for training the transformation layers.

Experimental Evaluation

The MFR method was evaluated across several diverse benchmarks:

  • Standard Vision: MNIST, CIFAR-10, and Mini-ImageNet.
  • Attribute-Based: CelebA face attributes (specifically HAIR, SKIN, and EYE features).
  • Cross-Domain: Repurposing a ResNet-18 (trained on ImageNet) for Diabetic Retinopathy detection in medical scans.

Results and Discussion

The MFR method consistently outperformed base models trained from scratch on few-shot data. A visualization of 120 different 3-way classification tasks highlights the broad applicability of the method, with substantial performance gains observed in the vast majority of cases.

Performance gain (green) and loss (red) across 120 tasks for MNIST, CIFAR-10, and Mini-ImageNet.

Key insights include:

  • Input as the Interface: Behavior can be altered by learning the "language" of a model's input space rather than changing its weights. This acts as a key to unlock latent feature-extraction power.
  • Heterogeneity as a Strength: Models with different output spaces can still provide complementary visual features. For example, a model trained on vehicles can provide useful features for distinguishing animal species.
  • Cross-Domain Transfer: The successful application to medical imaging demonstrates that general-purpose models contain latent knowledge applicable to highly specialized domains.

Conclusion and Future Work

While MFR presents a significant step forward in black-box model utilization, it introduces computational overhead due to the training of VAEs and Normalizing Flows. Furthermore, the reliance on Random Gradient Estimation requires multiple API calls per training step. Future research may focus on automating the selection of the most relevant black-box models from a library to further optimize performance and efficiency.

Summary updated

BibTeX
@inproceedings{hoang2024few,
  author = {Minh Hoang and Trong Nghia Hoang},
  title = {{Few-Shot Learning via Repurposing Ensemble of Black-Box Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29137},
}
Figure