ViM: Vision Middleware for Unified Downstream Transferring

Yutong Feng, Biao Gong, Jianwen Jiang, Yiliang Lv, Yujun Shen, Deli Zhao, Jingren Zhou

Introduction

The prevailing "pretrain-finetune" paradigm in computer vision has enabled significant leaps in performance. However, it faces a persistent challenge: task preference. Foundation models, such as CLIP or ImageNet-pretrained ViTs, often exhibit a bias toward their pre-training objectives. For instance, a model optimized for image-level classification may lack the localized spatial awareness required for dense tasks like semantic segmentation.

To bridge this "task gap," researchers typically resort to full fine-tuning—which is computationally expensive and results in fragmented, task-specific models—or multi-task pre-training, which is difficult to scale. This paper introduces ViM (Vision Middleware), a modular framework that utilizes a "zoo" of lightweight adapters to adapt a single frozen foundation model to diverse downstream applications.

The ViM Framework

ViM operates through a three-tier architecture designed to decouple general feature extraction from task-specific adaptation:

  1. Upstream (Foundation Model): A large-scale model (e.g., CLIP-B/16) provides the core representation. Crucially, this backbone remains frozen throughout the process, preserving its general knowledge and drastically reducing training overhead.
  2. Midstream (Module Zoo): This layer consists of a collection of lightweight adapters based on the ConvPass architecture. Each module is trained independently on a specific mid-scale task (e.g., object detection, scene parsing, or visual question answering). These modules act as "knowledge anchors," capturing specialized features that the foundation model might overlook. Because they represent only ~0.4% of the total parameters, storing a diverse zoo is highly efficient.
  3. Downstream (Adaptive Aggregation): For a specific downstream task, ViM adaptively aggregates multiple modules from the zoo. This allows the system to leverage complementary knowledge from different midstream tasks—for example, using detection-based knowledge to improve a segmentation task.

Adaptive Aggregation Strategies

The researchers explore two primary methods for combining the expertise of the module zoo:

  • Ensemble-based Aggregation: A routing mechanism assigns weights to different modules based on the input. The final output is a weighted combination of the modules' individual outputs.
  • Reparameterization-based Aggregation: To eliminate inference overhead, the weights of the selected adapters are merged into a single set of parameters before deployment. This ensures that the final model maintains the same inference speed as the original foundation model while benefiting from the aggregated knowledge.

Experimental Evaluation

The efficacy of ViM was tested across a broad spectrum of datasets, utilizing 47 sub-tasks for midstream training (including ImageNet-21K and Objects365) and 30 tasks for downstream evaluation.

Key Results

ViM demonstrates significant improvements over baseline methods across classification, detection, and segmentation. Using a CLIP-B/16 backbone, ViM achieved an average score of 79.6 across 30 tasks, compared to 76.3 for full fine-tuning and 77.8 for standard adapter tuning.

Key takeaways from the results include:

  • Balanced Performance: Unlike single-task fine-tuning, which often improves one metric at the expense of others, ViM provides consistent gains across disparate tasks simultaneously.
  • Efficiency and Scalability: By keeping the backbone frozen, ViM is significantly cheaper to train than full fine-tuning. Furthermore, performance scales positively with the size and diversity of the midstream module zoo.
  • Zero-Overhead Inference: Through reparameterization, ViM achieves state-of-the-art results without increasing the computational cost during deployment.

Conclusion and Future Work

ViM introduces a flexible, modular approach to vision model adaptation. By positioning a "middleware" layer between foundation models and downstream tasks, it overcomes the rigidity of traditional fine-tuning. The authors envision ViM as a community-driven project where researchers can contribute specialized modules to a global zoo.

Future work will focus on refining the "recipe" for module selection and extending the framework to multi-modal architectures beyond vision. ViM paves the way for a collaborative ecosystem where specialized knowledge can be shared and combined to solve increasingly complex visual tasks.

Summary updated

Abstract

Foundation models are pre-trained on massive data and transferred to downstream tasks via fine-tuning. This work presents Vision Middleware (ViM), a new learning paradigm that targets unified transferring from a single foundation model to a variety of downstream tasks. ViM consists of a zoo of lightweight plug-in modules, each of which is independently learned on a midstream dataset with a shared frozen backbone. Downstream tasks can then benefit from an adequate aggregation of the module zoo thanks to the rich knowledge inherited from midstream tasks. There are three major advantages of such a design. From the efficiency aspect, the upstream backbone can be trained only once and reused for all downstream tasks without tuning. From the scalability aspect, we can easily append additional modules to ViM with no influence on existing modules. From the performance aspect, ViM can include as many midstream tasks as possible, narrowing the task gap between upstream and downstream. Considering these benefits, we believe that ViM, which the community could maintain and develop together, would serve as a powerful tool to assist foundation models.
BibTeX
@inproceedings{feng2023vim,
  author = {Yutong Feng and Biao Gong and Jianwen Jiang and Yiliang Lv and Yujun Shen and Deli Zhao and Jingren Zhou},
  title = {{ViM: Vision Middleware for Unified Downstream Transferring}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure