Task-Specific Adaptation with Restricted Model Access

Matan Levy, Rami Ben-Ari, Dvir Samuel, Nir Darshan, Dani LischinskiSubmission to ICLR 2026 · Rejected

TL;DR

Current parameter-efficient fine-tuning (PEFT) techniques like LoRA alter internal model layers, complicating multi-task serving and risking proprietary weight leakage. This paper introduces Gray-box fine-tuning, an alternative that adapts completely frozen foundation models using external input/output adapters and optional intermediate proxy tokens. By leaving the internal architecture and computational flow entirely untouched, it enables high-throughput multi-task serving while achieving performance remarkably close to white-box baselines.

What Problem Does This Paper Address?

Adapting foundation models (e.g., CLIP, BLIP) to downstream tasks usually involves either Full Fine-Tuning or PEFT methods like LoRA. While PEFT requires training fewer parameters, it still modifies internal model weights or inserts auxiliary bottleneck layers into the transformer’s forward pass. This creates three major headaches:

  1. Deployment and Scalability: Serving hundreds of custom tasks requires dynamically loading, swapping, or routing distinct adapter weights, making unified batching across different tasks virtually impossible.
  2. Hardware and Edge Optimization: Modifying internal graphs breaks compilation optimizations (like operator fusion or fixed-graph pruning) across different edge targets.
  3. Security and IP Protection: Foundation model providers want to serve models without exposing weights. Prior work has shown that LoRA updates can actually be inverted to recover proprietary pre-trained weights.

Black-box tuning (e.g., training a linear probing head on output embeddings) avoids touching the model, but it lacks the expressiveness needed to handle domain shifts.

Methods

The authors propose two "Gray-box" settings that allow gradient backpropagation through the frozen backbone during training while leaving the model's forward inference flow completely sealed:

[Input x] ──> [Input Adapter A] ──> [Frozen Backbone F] ──> [Output Adapter B] ──> [Prediction]
  • DarkGray-Box Adaptation (DGA): The backbone model is completely hidden. Adaptation occurs strictly through lightweight external endpoints:
  • Visual Input Adapter: A lightweight 2D convolutional block operating directly in pixel space. Remarkably, it uses no non-linear activation functions—acting purely as a learned affine transformation on input images.
  • Textual Input Adapter: Directly optimizes just two tokens: one context token prepended to the sequence (without positional encoding) and one shift token added element-wise to each token embedding. This consumes only a single extra slot of limited context windows (e.g., CLIP’s 77-token cap).
  • Output Adapter: A single linear layer applied to the backbone’s final pooled embeddings.
  • LightGray-Box Adaptation (LGA): For scenarios where the model host permits intermediate access, LGA introduces $k$ learnable proxy vectors into transformer attention layers ($Q, K, V$). Crucially, after computing attention, the proxy tokens are discarded and not passed to subsequent layers. No new layers are inserted, preserving the native layer execution flow.

Datasets

The authors evaluated their framework across a broad suite of tasks:

  • Text-to-Image Retrieval: MS-COCO 5k (including 12 custom object-based sub-domain splits) and Flickr30K.
  • Fine-Grained Retrieval: Stanford-Cars (197 vehicle classes).
  • Text-to-Video Retrieval: MSR-VTT and VATEX (evaluated frame-by-frame).
  • Image Classification: ImageNet-1K (16-shot setting) and ImageNet-Sketch.
  • Cross-Domain Retrieval: The Sketchy dataset (sketch-to-natural-image matching).
  • Additional Benchmarks (Appendix): RSCID (satellite image generation via DiT), MRPC (language understanding via DeBERTa-v3), and image captioning on BLIP-2.

Key Insights and Ideas

  • Computational Flow Invariance: The core philosophy is not designing complex adapters, but keeping the backbone's execution graph 100% untouched. This allows a single model instance to process a heterogeneous batch containing queries for multiple different downstream tasks simultaneously—each sample simply runs through its respective lightweight pre/post-adapter.
  • Pixel-Space Steering Works: Contrary to the intuition that deep vision transformers require internal adaptation, a simple linear/affine transformation on raw pixels combined with an output linear probe provides substantial steerability.
  • Minimalist Prompting: Instead of learning dozens of prefix tokens (which eats context length and requires training projection MLPs), shifting the embedding space with two tokens is sufficient for retrieval tasks.

Key Results

  • Retrieval Parity: On COCO 5k (BLIP backbone), DGA reaches 53.18% R@1, trailing LoRA (53.48%) by only 0.30% while beating zero-shot (47.04%) by 6.14 points. LGA achieves 54.14% R@1, rivaling full fine-tuning (53.06%).
  • Outperforming LoRA on Small Benchmarks: On the Stanford-Cars retrieval benchmark, DGA (97.16% P@1) and LGA (98.45% P@1) significantly outperformed LoRA (90.08% P@1), suggesting input-space adaptation is less prone to overfitting when target class descriptions are constrained.
  • Serving Efficiency: When serving 10 different tasks over 1,000 samples, DGA reduced FLOPs from 203.2 GFLOPs (separate models) to 22.8 GFLOPs, and cut memory footprint from 14.54 GB to 1.46 GB.

Limitations and Future Work

  • Extreme Domain Gaps: On tasks with massive distribution shifts—such as Sketch-to-Image retrieval on Sketchy—DGA lags behind white-box tuning (31.20% R@1 vs. 58.72% for LoRA and 69.20% for Full FT). Modifying internal representations remains essential when the feature distribution diverges drastically from pre-training.
  • Architecture Constraints for LGA: LGA relies on injecting proxy tokens into self-attention layers, making it inapplicable to pure CNN backbones.
  • Theoretical Security Limits: While DGA conceals weights and architecture, gradients are still backpropagated to the input adapter. Formal bounds on whether model inversion or extraction attacks could exploit these gradient streams remain an open problem.

Summary updated

Abstract

Modern foundation models achieve state-of-the-art performance across diverse modalities, yet commonly require modification of internal weights or insertion of new layers during fine-tuning. Such modifications increase deployment complexity, hinder optimization for edge devices, and risk exposure of proprietary model parameters. In this paper we analyze for the first time existing fine-tuning paradigms in the context of these three axes. Within this context we introduce ``Gray-box'' fine-tuning: a lightweight and deployment-friendly framework that adapts frozen backbones without altering their architecture or internal parameters. Gray-box fine-tuning enables adaptation solely via compact, external input/output adapters trained with controlled gradient signals at predefined model entry points, preserving all internal components unchanged. We introduce two variants: DarkGray-Box Adaptation (DGA), restricting modifications strictly to input and output interfaces, and LightGray-Box Adaptation (LGA), allowing limited injection of learnable tokens at intermediate layers for enhanced adaptability. Extensive evaluations across tasks including text-to-image retrieval, video retrieval, image classification, sketch retrieval, and diffusion-based generation demonstrate that Gray-box methods achieve competitive performance relative to standard fine-tuning, despite significantly stricter constraints. By decoupling task-specific adaptation from internal model modifications, Gray-box fine-tuning provides an efficient, scalable, and secure alternative to conventional fine-tuning methods.
BibTeX
@misc{levy2025task,
  author = {Matan Levy and Rami Ben-Ari and Dvir Samuel and Nir Darshan and Dani Lischinski},
  title = {{Task-Specific Adaptation with Restricted Model Access}},
  year = {2025},
  eprint = {2502.00796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.00796},
}
Figure