Task-Specific Adaptation with Restricted Model Access

Matan Levy, Rami Ben-Ari, Dvir Samuel, Nir Darshan, Dani Lischinski

Abstract

Fine-tuning large foundation models usually requires full access to their internal weights, which raises privacy, intellectual property (IP), and deployment cost concerns. This paper introduces a "Gray-box" fine-tuning framework that adapts models to new tasks using only lightweight input and output adapters, keeping the core model hidden and frozen. Remarkably, this restricted approach achieves performance competitive with popular white-box methods like LoRA, while enabling massive memory and compute savings during multi-task deployment.

Introduction: The Challenge of Model Adaptation

Foundation models such as CLIP or DINOv2 are powerful tools, but adapting them to specific downstream tasks typically necessitates "White-box" fine-tuning methods like Full Fine-Tuning or LoRA. These methods require direct access to the model's architecture and internal weights, leading to several significant drawbacks:

  1. Deployment Bloat: Hosting separate fine-tuned models for every downstream task is prohibitively expensive in terms of storage and memory.
  2. IP and Privacy Risks: Proprietary weights are exposed, risking unauthorized use, model theft, or sensitive data extraction.
  3. Edge Optimization: Weight modifications require re-optimization for edge devices for every task.

Conversely, "Black-box" methods, which train only a linear classifier on final output features, protect the model but often suffer from poor performance. This research proposes a middle ground: effective adaptation without exposing internal model weights.

Proposed Gray-box Framework

The authors introduce a "Gray-box" paradigm with two variants based on the level of model access:

DarkGray-box (DGA)

In this setting, the model's internals are completely hidden. Users only have access to gradient flow at the input and output. Adaptation is achieved through lightweight adapters:

  • Visual Input Adapter: Utilizes simple 2D convolutional layers as an affine transformation on image pixels without non-linear activations.
  • Textual Input Adapter: Instead of long prompt prefixes, it optimizes just two tokens: an "extra" token appended to the sequence and a "shift" token added to all original input tokens.
  • Output Adapter: A simple linear layer applied to the model's output features.

LightGray-box (LGA)

LGA is a more relaxed setting where weights remain hidden, but learnable, task-specific "proxy" tokens can be injected into intermediate transformer layers to influence attention scores.

Overview of DarkGray-box (DGA) and LightGray-box (LGA) adaptation frameworks.
The Textual Input Adapter architecture using shift and context tokens.

Experimental Evaluation

The framework was evaluated across diverse tasks and backbones (CLIP, BLIP, DINOv2), including Text-to-Image retrieval (COCO, Flickr30K), Text-to-Video retrieval (MSR-VTT, VATEX), Image Classification (ImageNet-1K, ImageNet-Sketch), and specialized tasks like Sketch-to-Image retrieval.

Key Insights

  • Sufficiency of Input/Output Manipulation: Transforming the input and output spaces can rival internal fine-tuning for many standard tasks, proving that internal weight modification is not always necessary.
  • Efficiency in Text Adaptation: By using only two learned tokens, DGA preserves valuable context length, unlike traditional prompt-tuning methods that consume significant token budgets.
  • Multi-tasking Efficiency: Since the core backbone remains frozen, a single instance can serve multiple tasks simultaneously by swapping tiny adapters. This enables efficient batching across domains.

Results and Performance

  • Competitive Performance: On COCO Text-to-Image retrieval, DGA achieved a Recall@1 of 53.18%, trailing the LoRA baseline (53.48%) by only 0.30 points while significantly outperforming Zero-Shot (47.04%).
  • Resource Savings: In a 10-task simultaneous inference simulation, the Gray-box approach required only 1.46 GB of GPU memory and 22.7 GFLOPs, compared to 14.54 GB and 203.2 GFLOPs for hosting 10 separate fine-tuned backbones.
  • LGA vs. DGA: LGA consistently outperformed DGA due to intermediate layer access, though DGA remains preferable for strict privacy requirements.

Limitations and Future Work

The Gray-box approach is less effective under extreme domain shifts, such as Sketch-to-Image retrieval when using a model pre-trained on natural images. Additionally, while it hides weights, formal cryptographic verification against advanced model extraction attacks remains a subject for future research.

Summary updated

Abstract

Modern foundation models achieve state-of-the-art performance across diverse modalities, yet commonly require modification of internal weights or insertion of new layers during fine-tuning. Such modifications increase deployment complexity, hinder optimization for edge devices, and risk exposure of proprietary model parameters. In this paper we analyze for the first time existing fine-tuning paradigms in the context of these three axes. Within this context we introduce ``Gray-box'' fine-tuning: a lightweight and deployment-friendly framework that adapts frozen backbones without altering their architecture or internal parameters. Gray-box fine-tuning enables adaptation solely via compact, external input/output adapters trained with controlled gradient signals at predefined model entry points, preserving all internal components unchanged. We introduce two variants: DarkGray-Box Adaptation (DGA), restricting modifications strictly to input and output interfaces, and LightGray-Box Adaptation (LGA), allowing limited injection of learnable tokens at intermediate layers for enhanced adaptability. Extensive evaluations across tasks including text-to-image retrieval, video retrieval, image classification, sketch retrieval, and diffusion-based generation demonstrate that Gray-box methods achieve competitive performance relative to standard fine-tuning, despite significantly stricter constraints. By decoupling task-specific adaptation from internal model modifications, Gray-box fine-tuning provides an efficient, scalable, and secure alternative to conventional fine-tuning methods.
BibTeX
@misc{levy2025task,
  author = {Matan Levy and Rami Ben-Ari and Dvir Samuel and Nir Darshan and Dani Lischinski},
  title = {{Task-Specific Adaptation with Restricted Model Access}},
  year = {2025},
  eprint = {2502.00796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.00796},
}
Figure