TL;DR
The paper introduces ARMS, a lightweight (800M parameter) router designed to dynamically select the most suitable Vision-Language Model (VLM) for any given image-text query. By exploiting the "performance paradox"—the fact that smaller models often succeed where larger ones fail—and utilizing a novel multimodal dataset ($M^2$), ARMS effectively coordinates a pool of VLMs to outperform massive commercial models like GPT-4o.
What Problem Does This Paper Address?
Deploying Vision-Language Models (VLMs) presents a classic engineering dilemma: do you use a massive, expensive commercial API (like GPT-4o) or a smaller, free open-source model (like Qwen2.5-VL)?
Interestingly, larger models do not always outperform smaller ones across the board—a phenomenon known as the performance paradox. As shown in the heatmap below, open-source models frequently answer queries correctly that stump top-tier commercial models, and vice versa.
To exploit this complementarity, we need an intelligent "router" to send each query to the best-suited model. However, existing LLM routing methods fall short for VLMs due to three main challenges:
- Lack of specialized multimodal data for training routers.
- Ineffective feature representation that fails to blend image context with the specific capabilities of candidate VLMs.
- Rigid model spaces, meaning the router typically requires expensive full retraining whenever a new VLM is released.
Methods
The authors propose ARMS (an acronym for An effective Router for VLM Model Selection), which utilizes a Mixture-of-Experts (MoE) architecture to dynamically map queries to models.
1. Router Architecture
- Dual Encoders: ARMS uses DistilBERT to encode the text query alongside a structured VLM profile (text describing the candidate model's name, parameter count, and architecture). A CLIP ViT-B/32 model encodes the input image.
- Gate Network & Fusion Experts: A gating mechanism routes the text and image embeddings to specialized transformer-based Fusion Experts. These experts use cross-attention to align visual and textual semantics based on the query type.
- Cosine Similarity Classifier: Instead of standard classification heads, the final layer computes the cosine similarity between the fused query representation and the VLM profile embedding. This is normalized using an inverse hyperbolic tangent (
arctanh) function to generate stable confidence scores.
2. Adaptation Strategies for New VLMs
To avoid full retraining when new models enter the ecosystem, the authors introduce two strategies:
- Incremental Training: Continuing to train the existing router parameters using data from the newly introduced VLMs.
- Independent Training: Training a completely separate router for the new models and combining it with the original router via a threshold-based cascaded system (Algorithm 1).
Datasets
Because no large-scale routing dataset existed for multimodal tasks, the authors built $M^2$, the first dedicated dataset for VLM selection.
- Base Data: Consists of 56,107 unique image-text queries compiled from benchmarks like ALM-bench, MMT-Bench, and MME-RealWorld.
- Model Outputs: Contains the ground-truth correctness labels, latencies, and API costs for 7 mainstream VLMs (4 open-source: Qwen2.5-VL-7B, Idefics2-8b, Llama3.2-11b, SmolVLM2-2b; and 3 commercial: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Flash).
- Evaluation: Out-of-distribution (OOD) generalization is tested using 10,000 queries sampled from the VHELM benchmark.
Key Insights and Ideas
- VLM Profiles as Inductive Priors: Explicitly feeding text descriptions of a model's capacity (e.g., its architecture and size) into the router allows it to generalize to new models much better than using simple categorical IDs.
- Expert Specialization: Visualizing the routing space via t-SNE reveals that the MoE framework successfully clusters queries by benchmark and VLM capability ranges, proving that different experts learn to handle distinct types of multimodal reasoning.
- Cascades Beat Fine-Tuning for New Models: When expanding the model pool, the Independent Training strategy significantly outperforms Incremental Training. Creating a separate router prevents the severe data imbalance issues that occur when fine-tuning an existing router on a small set of new model samples.
Key Results
ARMS demonstrates strong performance across both in-distribution (ID) and out-of-distribution (OOD) settings using the Exact Match (EM) metric:
- Base Pool (4 Open-Source VLMs): ARMS achieves an EM score of 0.5739 (ID) and 0.4321 (OOD), outperforming the best single open-source model (Llama 3.2-11b at 0.5551 ID / 0.2699 OOD).
- Expanded Pool (7 VLMs via Cascade): When commercial models are added, the 800M ARMS router achieves an OOD EM score of 0.5428, noticeably outperforming a standalone GPT-4o (0.4346) and Claude 3.5 Sonnet (~0.15 on this specific slice).
Limitations and Future Work
While ARMS successfully optimizes for accuracy, the authors explicitly disregard inference latency and monetary cost in their routing objective function due to the difficulty of maintaining consistent hardware environments for open-source vs. cloud API models.
A natural next step for future work is extending the ARMS framework into a multi-objective optimization router that balances accuracy against user-defined latency budgets and token cost constraints.
Summary updated