Musketeer (All for One, and One for All): A Generalist Vision-Language Model with Task Explanation Prompts

Zhaoyang Zhang, Yantao Shen, Kunyu Shi, Zhaowei Cai, Jun Fang, Siqi Deng, Hao Yang, Davide Modolo, Zhuowen Tu, Stefano SoattoarXiv 2023 · DOI · arXiv

Introduction

In the evolution of Large Language Models (LLMs), a single set of parameters typically handles a vast array of tasks, from creative writing to code generation. However, in the domain of Vision-Language (VL) modeling, achieving such "generalist" behavior has proven difficult. Most current foundation models rely on task-specific "heads" or adapters to manage the heterogeneity of tasks like visual grounding (outputting coordinates) and visual question answering (answering text). These architectural silos increase complexity and prevent models from fully leveraging shared knowledge across different tasks.

The Musketeer framework addresses these challenges by introducing a truly unified generalist model. By employing Task Explanation Prompts (TEP), Musketeer eliminates the need for specialized architectural components, allowing a single set of parameters to excel across diverse vision and language benchmarks.

The Musketeer Framework

Task Explanation Prompts (TEP)

The central innovation of Musketeer is the Task Explanation Prompt. Unlike traditional "one-hot" prompts (which use a simple numerical ID to signal a task) or "base prompts" (short instructions like "summarize this"), a TEP provides a rich, structured natural language description of the task at hand.

Comparison of the structured Task Explanation Prompt (TEP) against standard one-hot and base prompt baselines.

As shown in Figure 1, a TEP consists of five distinct components:

  1. Data Description: Contextual information about the dataset's origin.
  2. Input Format: A summary of the multi-modal sequence structure.
  3. Output Format: Explicit specifications for the expected output (e.g., coordinate formats like "x0 + y0 + x1 + y1").
  4. Output Description: Detailed instructions on what the generated targets represent.
  5. Instance Prompt: The specific query for the current input.

Model Architecture

Musketeer utilizes a standard Transformer-based encoder-decoder architecture, initialized from the OFA model. The model processes image tokens and text tokens (which include the TEP) through a unified pipeline.

The Musketeer architecture, showing the integration of image tokens and TEP-enhanced text tokens into a unified transformer pipeline.

By providing the "instruction manual" within the input sequence itself, the model learns to instantiate internal processing pathways for specific tasks. This shift from architectural specialization to informational specialization allows the model to remain flexible and easily extendable to new tasks without modifying its core structure.

Experimental Setup and Results

The Musketeer model was evaluated across seven diverse benchmarks, including Image Captioning (COCO), Visual Grounding (RefCOCO), Visual Question Answering (VQAv2), and Text Summarization (Gigaword).

Performance and Multi-task Synergy

Musketeer demonstrates performance competitive with, and often superior to, task-specific specialist models. Notably, it outperforms larger multi-task frameworks like Unified-IO and Uni-Perceiver. For instance, the MusketeerLarge variant (472M parameters) surpassed the much larger Unified-IO_XLarge (2.9B parameters) on several key grounding and captioning metrics.

A critical finding is the "More is Better" effect. While traditional multi-tasking often suffers from task interference—where performance on one task degrades as others are added—Musketeer showed improved performance as the number of tasks increased from three to seven. This suggests that the TEP structure helps the model harvest synergies between related tasks, such as object detection and visual grounding, by using natural language as a conceptual bridge.

Generalization and Zero-Shot Capabilities

The use of natural language explanations significantly enhances the model's ability to generalize. In zero-shot evaluations on unseen tasks (e.g., training on everything except visual entailment), TEP-based models performed 10% better than those using standard prompts. This indicates that the structured explanations provide a logical framework that helps the model apply learned concepts to novel scenarios.

Conclusion and Future Work

Musketeer represents a significant step toward truly generalist vision-language models. By replacing task-specific heads with rich, natural language instructions, it achieves high performance while maintaining architectural simplicity.

However, the use of long TEP sequences introduces a latency overhead of approximately 10-15% during inference. Future research will focus on optimizing these prompts—perhaps through compression techniques—to maintain instructional benefits while improving computational efficiency. Additionally, the authors aim to apply the Musketeer approach to even more diverse model backbones and a wider range of multi-modal tasks.

Summary updated

Abstract

We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). With rich and structured information such as task input/output format, TEP reduces interference among tasks, allowing the model to focus on their shared structure. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.
BibTeX
@misc{zhang2023musketeer,
  author = {Zhaoyang Zhang and Yantao Shen and Kunyu Shi and Zhaowei Cai and Jun Fang and Siqi Deng and Hao Yang and Davide Modolo and Zhuowen Tu and Stefano Soatto},
  title = {{Musketeer (All for One, and One for All): A Generalist Vision-Language Model with Task Explanation Prompts}},
  year = {2023},
  eprint = {2305.07019},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.07019},
}
Figure