Abstract
Fine-tuning large foundation models usually requires full access to their internal weights, which raises privacy, intellectual property (IP), and deployment cost concerns. This paper introduces a "Gray-box" fine-tuning framework that adapts models to new tasks using only lightweight input and output adapters, keeping the core model hidden and frozen. Remarkably, this restricted approach achieves performance competitive with popular white-box methods like LoRA, while enabling massive memory and compute savings during multi-task deployment.
Introduction: The Challenge of Model Adaptation
Foundation models such as CLIP or DINOv2 are powerful tools, but adapting them to specific downstream tasks typically necessitates "White-box" fine-tuning methods like Full Fine-Tuning or LoRA. These methods require direct access to the model's architecture and internal weights, leading to several significant drawbacks:
- Deployment Bloat: Hosting separate fine-tuned models for every downstream task is prohibitively expensive in terms of storage and memory.
- IP and Privacy Risks: Proprietary weights are exposed, risking unauthorized use, model theft, or sensitive data extraction.
- Edge Optimization: Weight modifications require re-optimization for edge devices for every task.
Conversely, "Black-box" methods, which train only a linear classifier on final output features, protect the model but often suffer from poor performance. This research proposes a middle ground: effective adaptation without exposing internal model weights.
Proposed Gray-box Framework
The authors introduce a "Gray-box" paradigm with two variants based on the level of model access:
DarkGray-box (DGA)
In this setting, the model's internals are completely hidden. Users only have access to gradient flow at the input and output. Adaptation is achieved through lightweight adapters:
- Visual Input Adapter: Utilizes simple 2D convolutional layers as an affine transformation on image pixels without non-linear activations.
- Textual Input Adapter: Instead of long prompt prefixes, it optimizes just two tokens: an "extra" token appended to the sequence and a "shift" token added to all original input tokens.
- Output Adapter: A simple linear layer applied to the model's output features.
LightGray-box (LGA)
LGA is a more relaxed setting where weights remain hidden, but learnable, task-specific "proxy" tokens can be injected into intermediate transformer layers to influence attention scores.
Experimental Evaluation
The framework was evaluated across diverse tasks and backbones (CLIP, BLIP, DINOv2), including Text-to-Image retrieval (COCO, Flickr30K), Text-to-Video retrieval (MSR-VTT, VATEX), Image Classification (ImageNet-1K, ImageNet-Sketch), and specialized tasks like Sketch-to-Image retrieval.
Key Insights
- Sufficiency of Input/Output Manipulation: Transforming the input and output spaces can rival internal fine-tuning for many standard tasks, proving that internal weight modification is not always necessary.
- Efficiency in Text Adaptation: By using only two learned tokens, DGA preserves valuable context length, unlike traditional prompt-tuning methods that consume significant token budgets.
- Multi-tasking Efficiency: Since the core backbone remains frozen, a single instance can serve multiple tasks simultaneously by swapping tiny adapters. This enables efficient batching across domains.
Results and Performance
- Competitive Performance: On COCO Text-to-Image retrieval, DGA achieved a Recall@1 of 53.18%, trailing the LoRA baseline (53.48%) by only 0.30 points while significantly outperforming Zero-Shot (47.04%).
- Resource Savings: In a 10-task simultaneous inference simulation, the Gray-box approach required only 1.46 GB of GPU memory and 22.7 GFLOPs, compared to 14.54 GB and 203.2 GFLOPs for hosting 10 separate fine-tuned backbones.
- LGA vs. DGA: LGA consistently outperformed DGA due to intermediate layer access, though DGA remains preferable for strict privacy requirements.
Limitations and Future Work
The Gray-box approach is less effective under extreme domain shifts, such as Sketch-to-Image retrieval when using a model pre-trained on natural images. Additionally, while it hides weights, formal cryptographic verification against advanced model extraction attacks remains a subject for future research.
Summary updated