Abstract
This paper introduces a framework for "black-box repurposing," allowing researchers to solve new tasks with very little data by leveraging an ensemble of existing, frozen AI models. By learning to intelligently transform inputs to "prompt" these black-box APIs for useful features, the authors achieve high-performance few-shot learning without ever needing access to the models' internal weights or original training data.
Introduction
In the current deep learning landscape, leveraging pre-trained models typically involves transfer learning or meta-learning. However, these techniques generally require "white-box" access to internal parameters (weights) or the massive datasets used for initial training. In contrast, many state-of-the-art models are deployed as "Black-Box" APIs (Machine Learning-as-a-Service, or MLaaS), where internal architectures are hidden. This research addresses the challenge of using these restricted, heterogeneous models to solve new "target" tasks in few-shot settings, where only a handful of labeled examples are available.
Methodology: Model Fusion via Repurposing (MFR)
The proposed Model Fusion via Repurposing (MFR) framework avoids modifying black-box models directly. Instead, it focuses on modifying the input sent to them to elicit useful feature responses.
The framework consists of four primary components:
- Input Augmentation via Normalizing Flows: Invertible transformations (e.g., Real NVP) warp target task inputs into a format that triggers useful responses from the black-box models.
- Latent Factorization: A Variational Autoencoder (VAE) approach decomposes augmented inputs into a common factor (shared knowledge) and a specific factor (task-unique knowledge).
- Ensemble Integration: Outputs from multiple models are combined. This allows models trained on disparate domains (e.g., digits vs. animals) to contribute to a rich feature set for the new task.
- Random Gradient Estimation: To overcome the non-differentiable nature of black-box APIs, the authors use a finite difference trick, perturbing inputs to estimate the gradients required for training the transformation layers.
Experimental Evaluation
The MFR method was evaluated across several diverse benchmarks:
- Standard Vision: MNIST, CIFAR-10, and Mini-ImageNet.
- Attribute-Based: CelebA face attributes (specifically HAIR, SKIN, and EYE features).
- Cross-Domain: Repurposing a ResNet-18 (trained on ImageNet) for Diabetic Retinopathy detection in medical scans.
Results and Discussion
The MFR method consistently outperformed base models trained from scratch on few-shot data. A visualization of 120 different 3-way classification tasks highlights the broad applicability of the method, with substantial performance gains observed in the vast majority of cases.
Key insights include:
- Input as the Interface: Behavior can be altered by learning the "language" of a model's input space rather than changing its weights. This acts as a key to unlock latent feature-extraction power.
- Heterogeneity as a Strength: Models with different output spaces can still provide complementary visual features. For example, a model trained on vehicles can provide useful features for distinguishing animal species.
- Cross-Domain Transfer: The successful application to medical imaging demonstrates that general-purpose models contain latent knowledge applicable to highly specialized domains.
Conclusion and Future Work
While MFR presents a significant step forward in black-box model utilization, it introduces computational overhead due to the training of VAEs and Normalizing Flows. Furthermore, the reliance on Random Gradient Estimation requires multiple API calls per training step. Future research may focus on automating the selection of the most relevant black-box models from a library to further optimize performance and efficiency.
Summary updated