Adapting Foundation Models Under Black Box Constraints

522 concepts358 papers2 proposals

The paradigm of Model-as-a-Service (MaaS) has fundamentally changed how we interact with state-of-the-art artificial intelligence. Instead of downloading model weights to local hardware, users increasingly query proprietary models hosted behind restricted APIs. As noted by Taylor Sorensen et al. in An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels, this lack of transparency creates a hard boundary: it completely precludes the use of standard gradient-based optimization or soft-prompt tuning.

When internal parameters, hidden states, and gradients are hidden behind an API wall, traditional backpropagation breaks down. To specialize these "black-box" models for downstream tasks, researchers have developed a new class of adaptation techniques. These methods operate entirely on the periphery, optimizing inputs, calibrating outputs, or estimating gradients externally without ever modifying a single internal weight.


How do you guide a model toward a specific behavior when you cannot compute a gradient? The core intuition is to treat the prompt itself as a discrete decision variable, and the black-box model as an environment that returns an external reward or validation score. Instead of backpropagating errors through the architecture, we can use derivative-free search algorithms to iteratively mutate and select text prompts based on external performance.

[Candidate Prompt] ---> ( Black-Box Model ) ---> [Output / Score]
         ^                                              |
         |____________ [Search / Mutation Engine] <_____|

This search can be performed using simple heuristic edits. For instance, Archiki Prasad et al. developed GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models, which frames prompt optimization as a pure search problem. By performing text-level operations—such as deleting, swapping, or adding clauses—and evaluating the candidate instructions against a validation set, the authors show that greedy or beam search can effectively navigate the prompt space without gradient access.

Rather than relying on manual editing rules, we can also leverage language models themselves to drive the search. Yujian Betterest Li et al. proposed SPELL: Semantic Prompt Evolution based on a LLM, which treats an LLM as an evolutionary engine that automatically generates, mutates, and refines prompt candidates through a process of reproduction and selection.

Alternatively, reinforcement learning can navigate this massive token space by treating the target model as a frozen environment. In RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning, Mingkai Deng demonstrates that a tiny external policy network can learn to generate highly effective discrete prompts using only the output scores or performance metrics of the target black-box model as a reward signal.


Output-Space Calibration and External Reranking

Modifying the input prompt can be highly sensitive to minor formatting changes. A complementary strategy is to accept the black-box model's raw outputs as they are, and adapt them in the output space. If an API provides token probabilities (logits) or embeddings, we can build lightweight post-processing layers to calibrate or filter the results.

[Input] ---> ( Black-Box Model ) ---> [Raw Logits/Embeddings] ---> ( External Adapter ) ---> [Final Prediction]

Consider the challenge of prompt bias, where an in-context learning model might favor certain labels purely based on the order of examples. Momin Abbas et al. introduced Enhancing In-context Learning via Linear Probe Calibration, demonstrating that a simple linear transformation can calibrate an LLM's output probabilities using just a handful of labeled samples. This approach operates solely on the output probabilities, bypassing the need for internal parameter modifications while significantly stabilizing predictions.

When full personalization or task specialization is required, researchers often append external "scoring" or "reranking" models to filter the black-box outputs. Bowen Tan et al. designed Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer, a lightweight external model that acts as a high-performance scorer to inject task-specific knowledge by reranking candidate generations.

Similarly, to personalize outputs without touching base weights, Bo Dai et al. introduced HYDRA: Model Factorization Framework for Black-Box LLM Personalization. This framework pairs a shared black-box base model with user-specific external heads to rerank historical context, capturing individual user preferences entirely outside the model's core architecture.


Zeroth-Order Optimization for Continuous Adaptation

When task adaptation requires updating continuous parameters—such as the weights of an external input wrapper or a continuous visual prompt—but the base model remains a strict black box, researchers turn to Zeroth-Order (ZO) optimization. ZO methods approximate gradients by evaluating differences in forward-pass losses under random directional perturbations, allowing parameter updates without backpropagation.

Standard Backprop (White-Box):  [Loss] ---> Backpropagate through Layers ---> [Gradients]
Zeroth-Order (Black-Box):       [Loss(W + ε)] vs [Loss(W - ε)] -------------> [Estimated Gradients]

This forward-only approach makes continuous adaptation feasible in "gradient-banned" environments. In ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think, Tao Feng et al. demonstrate that models can achieve competitive adaptation accuracy using only forward passes, drastically reducing memory overhead and bypassing API restrictions.

This mechanism is also highly effective for distributed and collaborative settings. Yao Zhu et al. presented ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning, which combines zeroth-order optimization with input surgery and output re-mapping. This allows distributed clients to collaboratively tune adapters for black-box foundation models without ever requiring access to the centralized model's internal weights or gradients.

By shifting the optimization problem from the internal architecture to the external input and output spaces, black-box adaptation ensures that foundation models remain highly customizable, even when locked behind a proprietary API.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_black_box_model_adaptation_cd47af70,
  title = {black-box model adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-model-adaptation/rdr_cd47af70b47547e798035b76dea598a3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_black_box_model_adaptation_cd47af70_v1,
  title = {black-box model adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-model-adaptation/rdr_cd47af70b47547e798035b76dea598a3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake