SilVar: Speech-Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization

Tan-Hanh Pham, Le Hoang Nam, Phu-Vinh Nguyen, Chong‐Wah Ngo, Truong Son Hy2025 · DOI

Abstract

SilVar is the first open-source, end-to-end multimodal model designed to interpret images through spoken instructions. By integrating CLIP, Whisper, and LLaMA 3.1, it enables complex visual reasoning and precise object localization triggered directly by human speech. This work addresses the gap in current Visual Language Models (VLMs) that rely almost exclusively on text, offering a more natural interface for hands-busy or augmented reality environments.

Introduction

While text-based VLMs like LLaVA have achieved significant success in scene description and question answering, they lack the auditory grounding necessary for real-world human-machine interaction. Existing speech-LLMs typically focus on audio-to-text tasks without visual context. SilVar (Speech-driven Instruction Localizer and Visual Analyzer) bridges this gap by processing visual and auditory inputs simultaneously to perform high-level reasoning, such as identifying objects based on function or relational context rather than simple nomenclature.

Architecture and Methodology

The SilVar architecture is modular, utilizing a LLaMA 3.1-8B model as its central reasoning engine. It incorporates two specialized "senses":

  • Audio Encoder: A lightweight Whisper-tiny model extracts features from spoken instructions.
  • Visual Encoder: A CLIP (ViT-B/32) model processes input images.
  • Projectors: Linear layers and Transformer-based adapters map these diverse features into a shared embedding space compatible with the LLM.
High-level overview of the SilVar architecture, showing the integration of audio and visual inputs into a shared embedding space for the LLM.

The model is trained via a two-stage pipeline. First, Speech-to-Text Alignment ensures the audio encoder can map spoken instructions to textual equivalents. Second, Multimodal Instruction Fine-tuning trains the system on combined image-speech-text datasets to perform reasoning and localization tasks.

Benchmarking Speech-Driven Reasoning

To evaluate the model, the researchers introduced SilVar-bench, a dataset containing 4,161 speech-driven samples based on COCO images. The benchmark categorizes tasks into three levels of reasoning: Conversational (natural dialogue), Simple Reasoning (step-by-step logic), and Complex Reasoning (context-heavy scenarios).

Examples of the three reasoning levels (Conversational, Simple, and Complex) and the resulting object localization in SilVar-bench.

Additionally, the authors augmented existing text-based benchmarks—ScienceQA, MMMU, and LISA—by converting their instructions into speech using high-quality Text-to-Speech (TTS) engines, creating a robust multimodal training corpus.

Key Results and Insights

SilVar demonstrates competitive performance across several benchmarks, often nearing the accuracy of text-based counterparts:

  • MMMU Benchmark: SilVar achieved a score of 30.2 with speech instructions, compared to 31.8 with text.
  • ScienceQA: The model reached 63.21% accuracy on speech instructions.
  • Object Localization: On complex reasoning tasks, SilVar achieved an Intersection over Union (IoU) accuracy of 27.56%, successfully translating spoken descriptions into precise bounding boxes.

Analysis revealed a "modality gap," where spoken instructions—being inherently noisier than text—result in slightly lower performance. However, the model often performs better on "Complex Reasoning" tasks than "Simple" ones, suggesting that additional context helps the model triangulate visual features more effectively. Furthermore, replacing the LLaMA 3.1 base with a reasoning-specialized model like DeepSeek-R1-Distill-Llama-8B improved performance on MMMU from 30.4 to 32.8, indicating that the LLM's underlying reasoning capacity is a primary bottleneck.

Conclusion and Future Work

SilVar represents a significant step toward natural, speech-driven visual interaction. While currently limited by dataset size and occasional hallucinations, future work will focus on scaling training data and exploring end-to-end speech processing to further close the performance gap between spoken and typed instructions.

Summary updated

BibTeX
@misc{pham2025silvar,
  author = {Tan-Hanh Pham and Le Hoang Nam and Phu-Vinh Nguyen and Chong‐Wah Ngo and Truong Son Hy},
  title = {{SilVar: Speech-Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.589},
}
Figure