SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization

Tan-Hanh Pham, Hoang-Nam Le, Phu-Vinh Nguyen, Chris Ngo, Truong-Son HyarXiv 2024 · DOI · arXiv

Introduction

Recent advancements in Vision-Language Models (VLMs) have significantly enhanced our ability to interact with visual data through text. However, the reliance on text-based inputs presents a significant barrier in scenarios where hands-free interaction is required or text input is impractical. Furthermore, while Chain-of-Thought (CoT) reasoning has improved text-based visual understanding, its application to speech-driven instructions remains largely unexplored. This paper introduces SilVar, an end-to-end multimodal model designed to bridge this gap by enabling complex reasoning and precise object localization through conversational speech cues.

Methodology

SilVar utilizes a modular architecture that integrates specialized encoders with a powerful large language model (LLM) backbone. The system comprises three primary components:

  1. Audio Processing: A tiny Whisper variant (39M parameters) is employed to extract robust features from spoken instructions.
  2. Visual Processing: CLIP (ViT-B/32) serves as the visual backbone for image feature extraction.
  3. Reasoning Engine: LLaMA 3.1-8B acts as the central brain, processing projected multimodal tokens to generate textual responses and spatial coordinates.

To align these modalities, lightweight Multi-Layer Perceptron (MLP) adapters project 768-dimensional audio and visual tokens into the 4096-dimensional space of the LLM.

The SilVar architecture (left) and its two-stage training pipeline (right), involving speech-to-text alignment and multimodal instruction fine-tuning.

The training process is divided into two distinct stages:

  • Stage 1 (Speech-to-Text Alignment): Ensures the audio encoder and LLM are synchronized for basic speech recognition tasks.
  • Stage 2 (Multimodal Instruction Fine-tuning): The model is trained to simultaneously process images and speech to perform complex reasoning and generate bounding boxes ([BBOX]) for object localization.

Datasets

The model was evaluated using standard benchmarks such as ScienceQA and MMMU. Additionally, the researchers introduced SilVar-bench, a specialized dataset derived from 998 COCO 2014 images. Using GPT-4, three levels of reasoning questions (Conversation, Simple, and Complex) were generated for each image and converted into speech using Google Cloud’s Text-to-Speech API, featuring over 50 diverse voices. The LISA dataset was also utilized to refine reasoning-based segmentation and localization.

Results and Discussion

SilVar demonstrates state-of-the-art performance for speech-driven models. On the MMMU benchmark, SilVar achieved a score of 30.2 using speech instructions, closely trailing its text-based performance of 31.8. This minimal performance gap validates the efficiency of the audio-to-LLM projection.

A key finding is that SilVar performs better on "Complex Reasoning" tasks than on simpler conversational ones. This suggests that the additional context provided in descriptive speech prompts helps the model more effectively narrow its visual search space.

SilVar demonstrating spatial reasoning and object localization by identifying specific items based on descriptive verbal queries.

In terms of localization, SilVar achieved an Intersection over Union (IoU) accuracy of 23.31% for complex speech instructions on SilVar-bench. Unlike many closed-source models that only identify objects by name, SilVar provides both a detailed explanation of its reasoning and the exact spatial coordinates of the target objects.

Conclusion

SilVar represents a significant step toward more natural human-AI interaction by enabling speech-driven visual reasoning and localization. While currently limited by its reliance on synthetic speech and a relatively small benchmark dataset, future work will focus on handling real-world audio environments, including background noise and diverse accents, as well as expanding the model's capabilities toward streaming audio and general environmental sound understanding.

Summary updated

Abstract

Visual Language Models have demonstrated remarkable capabilities across tasks, including visual question answering and image captioning. However, most models rely on text-based instructions, limiting their effectiveness in human-machine interactions. Moreover, the quality of language models depends on reasoning and prompting techniques, such as COT, which remain underexplored when using speech instructions. To address these challenges, we propose SilVar, a novel end-to-end multimodal model that uses speech instructions for reasoning in visual question answering. In addition, we investigate reasoning techniques with levels including conversational, simple, and complex speech instruction. SilVar is built upon CLIP, Whisper, and LLaMA 3.1-8B, enabling intuitive interactions by allowing users to provide verbal or text instructions. To this end, we introduce a dataset designed to challenge models with speech-based reasoning tasks for object localization. This dataset enhances the model ability to process and explain visual scenes from spoken input, moving beyond object recognition to reasoning-based interactions. The experiments show that SilVar achieves SOTA performance on the MMMU and ScienceQA benchmarks despite the challenge of speech-based instructions. We believe SilVar will inspire next-generation multimodal reasoning models, toward expert artificial general intelligence. Our code and dataset are available here.
BibTeX
@misc{pham2024silvar,
  author = {Tan-Hanh Pham and Hoang-Nam Le and Phu-Vinh Nguyen and Chris Ngo and Truong-Son Hy},
  title = {{SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization}},
  year = {2024},
  eprint = {2412.16771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.16771},
}
Figure