Abstract
SilVar is the first open-source, end-to-end multimodal model designed to interpret images through spoken instructions. By integrating CLIP, Whisper, and LLaMA 3.1, it enables complex visual reasoning and precise object localization triggered directly by human speech. This work addresses the gap in current Visual Language Models (VLMs) that rely almost exclusively on text, offering a more natural interface for hands-busy or augmented reality environments.
Introduction
While text-based VLMs like LLaVA have achieved significant success in scene description and question answering, they lack the auditory grounding necessary for real-world human-machine interaction. Existing speech-LLMs typically focus on audio-to-text tasks without visual context. SilVar (Speech-driven Instruction Localizer and Visual Analyzer) bridges this gap by processing visual and auditory inputs simultaneously to perform high-level reasoning, such as identifying objects based on function or relational context rather than simple nomenclature.
Architecture and Methodology
The SilVar architecture is modular, utilizing a LLaMA 3.1-8B model as its central reasoning engine. It incorporates two specialized "senses":
- Audio Encoder: A lightweight Whisper-tiny model extracts features from spoken instructions.
- Visual Encoder: A CLIP (ViT-B/32) model processes input images.
- Projectors: Linear layers and Transformer-based adapters map these diverse features into a shared embedding space compatible with the LLM.
The model is trained via a two-stage pipeline. First, Speech-to-Text Alignment ensures the audio encoder can map spoken instructions to textual equivalents. Second, Multimodal Instruction Fine-tuning trains the system on combined image-speech-text datasets to perform reasoning and localization tasks.
Benchmarking Speech-Driven Reasoning
To evaluate the model, the researchers introduced SilVar-bench, a dataset containing 4,161 speech-driven samples based on COCO images. The benchmark categorizes tasks into three levels of reasoning: Conversational (natural dialogue), Simple Reasoning (step-by-step logic), and Complex Reasoning (context-heavy scenarios).
Additionally, the authors augmented existing text-based benchmarks—ScienceQA, MMMU, and LISA—by converting their instructions into speech using high-quality Text-to-Speech (TTS) engines, creating a robust multimodal training corpus.
Key Results and Insights
SilVar demonstrates competitive performance across several benchmarks, often nearing the accuracy of text-based counterparts:
- MMMU Benchmark: SilVar achieved a score of 30.2 with speech instructions, compared to 31.8 with text.
- ScienceQA: The model reached 63.21% accuracy on speech instructions.
- Object Localization: On complex reasoning tasks, SilVar achieved an Intersection over Union (IoU) accuracy of 27.56%, successfully translating spoken descriptions into precise bounding boxes.
Analysis revealed a "modality gap," where spoken instructions—being inherently noisier than text—result in slightly lower performance. However, the model often performs better on "Complex Reasoning" tasks than "Simple" ones, suggesting that additional context helps the model triangulate visual features more effectively. Furthermore, replacing the LLaMA 3.1 base with a reasoning-specialized model like DeepSeek-R1-Distill-Llama-8B improved performance on MMMU from 30.4 to 32.8, indicating that the LLM's underlying reasoning capacity is a primary bottleneck.
Conclusion and Future Work
SilVar represents a significant step toward natural, speech-driven visual interaction. While currently limited by dataset size and occasional hallucinations, future work will focus on scaling training data and exploring end-to-end speech processing to further close the performance gap between spoken and typed instructions.
Summary updated