Introduction
Recent advancements in Vision-Language Models (VLMs) have significantly enhanced our ability to interact with visual data through text. However, the reliance on text-based inputs presents a significant barrier in scenarios where hands-free interaction is required or text input is impractical. Furthermore, while Chain-of-Thought (CoT) reasoning has improved text-based visual understanding, its application to speech-driven instructions remains largely unexplored. This paper introduces SilVar, an end-to-end multimodal model designed to bridge this gap by enabling complex reasoning and precise object localization through conversational speech cues.
Methodology
SilVar utilizes a modular architecture that integrates specialized encoders with a powerful large language model (LLM) backbone. The system comprises three primary components:
- Audio Processing: A tiny Whisper variant (39M parameters) is employed to extract robust features from spoken instructions.
- Visual Processing: CLIP (ViT-B/32) serves as the visual backbone for image feature extraction.
- Reasoning Engine: LLaMA 3.1-8B acts as the central brain, processing projected multimodal tokens to generate textual responses and spatial coordinates.
To align these modalities, lightweight Multi-Layer Perceptron (MLP) adapters project 768-dimensional audio and visual tokens into the 4096-dimensional space of the LLM.
The training process is divided into two distinct stages:
- Stage 1 (Speech-to-Text Alignment): Ensures the audio encoder and LLM are synchronized for basic speech recognition tasks.
- Stage 2 (Multimodal Instruction Fine-tuning): The model is trained to simultaneously process images and speech to perform complex reasoning and generate bounding boxes (
[BBOX]) for object localization.
Datasets
The model was evaluated using standard benchmarks such as ScienceQA and MMMU. Additionally, the researchers introduced SilVar-bench, a specialized dataset derived from 998 COCO 2014 images. Using GPT-4, three levels of reasoning questions (Conversation, Simple, and Complex) were generated for each image and converted into speech using Google Cloud’s Text-to-Speech API, featuring over 50 diverse voices. The LISA dataset was also utilized to refine reasoning-based segmentation and localization.
Results and Discussion
SilVar demonstrates state-of-the-art performance for speech-driven models. On the MMMU benchmark, SilVar achieved a score of 30.2 using speech instructions, closely trailing its text-based performance of 31.8. This minimal performance gap validates the efficiency of the audio-to-LLM projection.
A key finding is that SilVar performs better on "Complex Reasoning" tasks than on simpler conversational ones. This suggests that the additional context provided in descriptive speech prompts helps the model more effectively narrow its visual search space.
In terms of localization, SilVar achieved an Intersection over Union (IoU) accuracy of 23.31% for complex speech instructions on SilVar-bench. Unlike many closed-source models that only identify objects by name, SilVar provides both a detailed explanation of its reasoning and the exact spatial coordinates of the target objects.
Conclusion
SilVar represents a significant step toward more natural human-AI interaction by enabling speech-driven visual reasoning and localization. While currently limited by its reliance on synthetic speech and a relatively small benchmark dataset, future work will focus on handling real-world audio environments, including background noise and diverse accents, as well as expanding the model's capabilities toward streaming audio and general environmental sound understanding.
Summary updated