An Effective Router for Vision-Language Model Selection

Can Wang, Shengwei Wang, Bolin Zhang, Zhiying Tu, Dianhui ChuarXiv 2026 · arXiv

TL;DR

The paper introduces ARMS, a lightweight (800M parameter) router designed to dynamically select the most suitable Vision-Language Model (VLM) for any given image-text query. By exploiting the "performance paradox"—the fact that smaller models often succeed where larger ones fail—and utilizing a novel multimodal dataset ($M^2$), ARMS effectively coordinates a pool of VLMs to outperform massive commercial models like GPT-4o.

What Problem Does This Paper Address?

Deploying Vision-Language Models (VLMs) presents a classic engineering dilemma: do you use a massive, expensive commercial API (like GPT-4o) or a smaller, free open-source model (like Qwen2.5-VL)?

Interestingly, larger models do not always outperform smaller ones across the board—a phenomenon known as the performance paradox. As shown in the heatmap below, open-source models frequently answer queries correctly that stump top-tier commercial models, and vice versa.

Figure 1: The proportion(%) of samples answered incorrectly by the row model but correctly by the column model on the VHELM [15] benchmark, showing the performance paradox between VLMs.

To exploit this complementarity, we need an intelligent "router" to send each query to the best-suited model. However, existing LLM routing methods fall short for VLMs due to three main challenges:

  1. Lack of specialized multimodal data for training routers.
  2. Ineffective feature representation that fails to blend image context with the specific capabilities of candidate VLMs.
  3. Rigid model spaces, meaning the router typically requires expensive full retraining whenever a new VLM is released.

Methods

The authors propose ARMS (an acronym for An effective Router for VLM Model Selection), which utilizes a Mixture-of-Experts (MoE) architecture to dynamically map queries to models.

Figure 3: Architecture of ARMS, consists of five modules: a text encoder, an image encoder, a gate module, fusion experts, and a classifier.

1. Router Architecture

  • Dual Encoders: ARMS uses DistilBERT to encode the text query alongside a structured VLM profile (text describing the candidate model's name, parameter count, and architecture). A CLIP ViT-B/32 model encodes the input image.
  • Gate Network & Fusion Experts: A gating mechanism routes the text and image embeddings to specialized transformer-based Fusion Experts. These experts use cross-attention to align visual and textual semantics based on the query type.
  • Cosine Similarity Classifier: Instead of standard classification heads, the final layer computes the cosine similarity between the fused query representation and the VLM profile embedding. This is normalized using an inverse hyperbolic tangent (arctanh) function to generate stable confidence scores.

2. Adaptation Strategies for New VLMs

To avoid full retraining when new models enter the ecosystem, the authors introduce two strategies:

  • Incremental Training: Continuing to train the existing router parameters using data from the newly introduced VLMs.
  • Independent Training: Training a completely separate router for the new models and combining it with the original router via a threshold-based cascaded system (Algorithm 1).

Datasets

Because no large-scale routing dataset existed for multimodal tasks, the authors built $M^2$, the first dedicated dataset for VLM selection.

  • Base Data: Consists of 56,107 unique image-text queries compiled from benchmarks like ALM-bench, MMT-Bench, and MME-RealWorld.
  • Model Outputs: Contains the ground-truth correctness labels, latencies, and API costs for 7 mainstream VLMs (4 open-source: Qwen2.5-VL-7B, Idefics2-8b, Llama3.2-11b, SmolVLM2-2b; and 3 commercial: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Flash).
  • Evaluation: Out-of-distribution (OOD) generalization is tested using 10,000 queries sampled from the VHELM benchmark.

Key Insights and Ideas

  • VLM Profiles as Inductive Priors: Explicitly feeding text descriptions of a model's capacity (e.g., its architecture and size) into the router allows it to generalize to new models much better than using simple categorical IDs.
  • Expert Specialization: Visualizing the routing space via t-SNE reveals that the MoE framework successfully clusters queries by benchmark and VLM capability ranges, proving that different experts learn to handle distinct types of multimodal reasoning.
  • Cascades Beat Fine-Tuning for New Models: When expanding the model pool, the Independent Training strategy significantly outperforms Incremental Training. Creating a separate router prevents the severe data imbalance issues that occur when fine-tuning an existing router on a small set of new model samples.

Key Results

ARMS demonstrates strong performance across both in-distribution (ID) and out-of-distribution (OOD) settings using the Exact Match (EM) metric:

  • Base Pool (4 Open-Source VLMs): ARMS achieves an EM score of 0.5739 (ID) and 0.4321 (OOD), outperforming the best single open-source model (Llama 3.2-11b at 0.5551 ID / 0.2699 OOD).
  • Expanded Pool (7 VLMs via Cascade): When commercial models are added, the 800M ARMS router achieves an OOD EM score of 0.5428, noticeably outperforming a standalone GPT-4o (0.4346) and Claude 3.5 Sonnet (~0.15 on this specific slice).

Limitations and Future Work

While ARMS successfully optimizes for accuracy, the authors explicitly disregard inference latency and monetary cost in their routing objective function due to the difficulty of maintaining consistent hardware environments for open-source vs. cloud API models.

A natural next step for future work is extending the ARMS framework into a multi-objective optimization router that balances accuracy against user-defined latency budgets and token cost constraints.

Summary updated

Abstract

Vision-language models (VLMs) with varying performance and resource requirements are widely deployed, making it difficult for users to select the most appropriate one among numerous VLM candidates. Existing work reveals the performance paradox phenomenon in language models and focuses on routing methods to solve it. However, developing a router for VLM selection is still a critical yet challenging problem, which primarily faces: 1) lack of specialized data, 2) ineffective feature representation, and 3) rigid model space and costly adaptation. In this paper, we construct a multimodal dataset for VLM selection, containing the outputs of seven mainstream VLMs on 32,626 unique image-text queries. We then propose ARMS, a router for VLM selection. ARMS enhances input signals with VLM profiles, employs a simple but effective architecture to improve representations of queries and VLM capabilities. To improve ARMS' adaptation to new VLMs, we propose two extension training strategies: incremental training and independent training. Experimental results on both in-distribution and out-of-distribution test sets demonstrate the effectiveness of ARMS. In particular, using our training strategy, ARMs (only 800M in size) can adapt to a broader VLM space and defeat commercial models like GPT-4o that are hundreds of times larger in scale. Our code, models, and datasets are available in the anonymous repository.
BibTeX
@misc{wang2026effective,
  author = {Can Wang and Shengwei Wang and Bolin Zhang and Zhiying Tu and Dianhui Chu},
  title = {{An Effective Router for Vision-Language Model Selection}},
  year = {2026},
  eprint = {2606.08970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.08970},
}

Loading related content...

Figure