VITA: Towards Open-Source Interactive Omni Multimodal LLM

Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, Haoyu Cao, Di Yin, Long Ma, Xiawu Zheng, Rongrong Ji, Yunsheng Wu, Ran He, Caifeng Shan, Xing Sun

TL;DR

VITA is the first open-source omni-multimodal large language model capable of simultaneously processing video, image, text, and audio while delivering a highly interactive, real-time user experience. By introducing specialized state tokens and a novel duplex deployment pipeline, VITA overcomes traditional interaction barriers to support advanced features like wake-word-free activation and real-time audio interruption.

What Problem Does This Paper Address?

While proprietary models like GPT-4o have revolutionized human-computer interaction by processing text, vision, and audio natively and end-to-end, the open-source community has lagged behind. Most open-source multimodal models focus strictly on image-text pairs and operate purely on a sequential, turn-based interaction model.

Traditional voice assistants suffer from two major limitations:

  1. Awakening Necessity: They require an explicit wake-word (e.g., "Hi Siri") or a button press to trigger input tracking.
  2. Uninterruptible Execution: The system blocks user input while generating a response, failing to handle natural human interruptions.

VITA bridges this gap by offering an open-source architecture designed for natural, multi-turn, omni-multimodal human-computer interaction.

Figure 1: Interaction of VITA. Traditional audio interaction requires a pre-defined wake-up word, e.g., saying “Hi! Siri∼” every time you ask a question, or a button to control the input question audio (Limitation 1). The human-computer interaction is always blocked when the model generates output, as the previous system can only respond to input queries sequentially (Limitation 2). By contrast, on the one hand, unlike previous methods where audio, text, and vision are always separated, VITA supports these modalities end-to-end. On the other hand, VITA makes two contributions to multimodal interaction. Non-awakening Interaction: VITA automatically filters background noise like non-query human voices, thereby eliminating the need for the wake-up word and the button. Audio Interrupt Interaction: If the user interrupts with another question, the generation process is paused, and the model immediately responds to the latest query.

Methods

VITA uses a three-stage training pipeline built on top of the Mixtral 8×7B Mixture-of-Experts (SMoE) architecture:

  1. LLM Instruction Tuning: The authors expanded Mixtral’s vocabulary with Chinese tokens (increasing the size from 32,000 to 51,747) and performed bilingual instruction tuning on a 5-million synthetic corpus to ensure strong language proficiency.
  2. Multimodal Alignment: Individual encoders for vision (InternViT-300M-448px) and audio (a 341M parameter Transformer-based encoder) are connected to the LLM using simple two-layer MLP connectors to align disparate feature spaces.
  3. Multimodal Instruction Tuning & State Tokens: To handle interaction end-to-end, the authors introduced three unique state tokens embedded at the beginning of responses:
  4. <1>: Identifies effective user query audio, prompting a full response.
  5. <2>: Identifies environmental noise or non-query human speech, signaling the model to output an end-of-sequence (EOS) token and remain silent.
  6. <3>: Identifies text-based queries.
Figure 3: Training pipeline of VITA. The first stage LLM Instruction Tuning enhances the language model Mixtral 8×7B by expanding its vocabulary size and fine-tuning it with a high-quality bilingual text corpus, thereby achieving proficiency in both Chinese and English. The second stage Multimodal Alignment connects individual encoders with the LLM to process various modalities. By amassing a substantial collection of high-caliber multimodal data, we synchronize the text feature space with that of video, image, and audio. The last stage Multimodal Instruction Tuning allows the model to follow text or audio instructions to understand the image or video. A specially designed state token is used to distinguish the type of input query, facilitating subsequent multimodal human-computer interaction.

The Duplex Deployment Pipeline

To achieve real-time interruption, VITA introduces a duplex deployment scheme running two model instances concurrently:

  • Generation Model: Actively processes the current user query and streams the response.
  • Monitoring Model: Continuously analyzes environmental audio. If the user interrupts with a new valid query (token <1>), the monitoring model pauses the generation model, aggregates the historical context, and immediately responds to the new query—effectively swapping identities with the generator.
Figure 2: Architecture of VITA. VITA is capable of processing inputs in the form of pure text/audio, as well as video/image combined with text/audio. Besides, we set different state tokens for different query inputs. <1> corresponds to the effective query audio, such as “what is the biggest animal in the world?”, for which we expect a response from the model. <2> corresponds to the noisy audio, such as someone in the environment calls me to eat, for which we expect the model not to reply. <3> corresponds to the query text, i.e., the question given by the user in text form. During the training phase, we try to teach the model to automatically distinguish different input queries. During the deployment phase, with <2> we can implement non-awakening interaction. Based on this, we further introduce a duplex scheme for the audio interrupt interaction. Two models are running at the same time, where the generation model is responsible for handling user queries. When the generation model starts working, the other model monitors the environment. If the user interrupts with another effective audio query, the monitoring model aggregates the historical context to respond to the latest query, while the generation model is paused and tune to monitor, i.e., the two models swap identities.

Datasets

VITA was trained and evaluated on a vast collection of open-source and synthetic datasets:

  • Vision/Video: ShareGPT4V, Allava-Caption, LLaVA-150K, Lvis-Instruct, ScienceQA, Anyword-3M, Open-Chart, and ShareGemini.
  • Audio Alignment: Wenetspeech (10,000+ hours of Mandarin) and Gigaspeech (10,000 hours of English) for Automatic Speech Recognition (ASR), alongside Wavcaps for audio captioning.
  • Interaction Data: The authors used Text-to-Speech (TTS) tools to convert roughly half of the multimodal text questions into audio queries. They also synthesized 474K negative "noisy audio" samples from non-query text to train the model's noise-filtering capabilities.

Key Insights and Ideas

  • End-to-End Noise Filtering: Rather than relying on a separate, brittle voice activity detection system to filter out background conversations, VITA treats noise filtering as a semantic classification task inside the LLM via the <2> token.
  • Duplex Role-Swapping: Autoregressive generation makes real-time interruption difficult for a single model instance. Running twin models in a duplex pipeline elegantly bypasses this bottleneck, allowing instant interruption without losing conversation history.
  • Context Concatenation for Efficiency: To optimize training throughput, the authors concatenated text and image-text pairs up to a unified 6K token context length. This balanced token counts across training batches and significantly improved computational efficiency.
Figure 4: Data Concatenation. Different text data is directly concatenated to 6K tokens. Images are first divided into local patches and then different image-text pairs are concatenated. Video data is directly sampled frame by frame as input, with- out the need for concatenation. In this way, we could unify the length of data in different training batches, thus improving training efficiency.

Key Results

  • Language Upgrades: Vocabulary expansion and bilingual tuning significantly boosted Mixtral's base capabilities, raising its C-EVAL score from 53.30 to 56.68 and its GSM8K math reasoning score from 63.99 to 75.66.
  • Multimodal Performance: On standard image benchmarks (like MME, MMBench, and MMMU), VITA demonstrated performance comparable to image-specialized open-source models like LLaVA-Next and closely approached proprietary models like Gemini 1.5 Pro.
Evaluation of VITA across various image and video understanding benchmarks.

Limitations and Future Work

Despite its pioneering architecture, the authors note several areas for improvement:

  1. The Proprietary Gap: A performance gap still remains between VITA and closed-source giants like GPT-4o, particularly in complex reasoning tasks.
  2. Noise Misclassification: VITA occasionally misclassifies background noise or passive environmental speech as an active query, indicating a need for more diverse negative training data.
  3. External TTS Dependency: VITA currently outputs text tokens that are subsequently converted to audio via an external TTS engine. Integrating native, end-to-end speech synthesis directly into the LLM is a key objective for future iterations to minimize interaction latency.

Summary updated

Abstract

The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research. Project Page: https://vita-home.github.io.
BibTeX
@misc{fu2024vita,
  author = {Chaoyou Fu and Haojia Lin and Zuwei Long and Yunhang Shen and Yuhang Dai and Meng Zhao and Yi-Fan Zhang and Shaoqi Dong and Yangze Li and Xiong Wang and Haoyu Cao and Di Yin and Long Ma and Xiawu Zheng and Rongrong Ji and Yunsheng Wu and Ran He and Caifeng Shan and Xing Sun},
  title = {{VITA: Towards Open-Source Interactive Omni Multimodal LLM}},
  year = {2024},
  eprint = {2408.05211},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.05211},
}
Figure