ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools

Shaofeng Yin, Ting Lei, Yang LiuICCV 2025

TL;DR

This paper introduces ToolVQA, a novel dataset designed to bridge the gap in Large Foundation Models' (LFMs) ability to perform multi-step reasoning using external tools in real-world visual question answering (VQA) scenarios. They also present ToolEngine, a data generation pipeline that creates these complex, realistic datasets, and demonstrate that fine-tuning LLaVA-7B on ToolVQA significantly improves its tool-use capabilities, even outperforming GPT-3.5-Turbo on out-of-distribution tasks.

What Problem Does This Paper Address?

Large Foundation Models (LFMs) are increasingly being integrated with external tools to enhance their problem-solving abilities. While this has shown promise in Visual Question Answering (VQA), existing benchmarks often fall short by using simplified, synthetic scenarios and queries that don't reflect real-world complexity. This leads to a gap in LFMs' proficiency in functionally diverse multimodal settings that require multi-step reasoning. Current datasets often lack realistic visual contexts and implicitly embed complex reasoning, making it difficult to accurately assess and improve LFMs' tool-use capabilities in scenarios that mimic human interaction.

Methods

The core of this work lies in ToolEngine, a novel data generation pipeline designed to create realistic, multi-step reasoning VQA data. ToolEngine employs:

  • Image-guided Depth-First Search (DFS) on a Tool Graph: This approach simulates human-like multi-step reasoning by exploring a graph of available tools.
  • LCS-based Example Matching: To guide the DFS, this mechanism dynamically matches existing real-world tool-use examples based on the current trajectory, ensuring richer information extraction and more challenging queries. This is a key innovation, moving beyond fixed example matching.
  • Real-World Example Construction: Human-constructed examples are used as prior knowledge to guide the LFM controller, ensuring that generated queries align with human needs and tool usage patterns.

The pipeline aims to generate data that reflects real-world scenarios (complex visual contexts) and real-world queries (implicit multi-step reasoning).

Datasets

  • ToolVQA: The primary contribution is this new, large-scale multimodal dataset comprising 23K samples. It features 10 multimodal tools across 7 diverse domains and an average inference length of 2.78 reasoning steps per sample. ToolVQA is designed to be more realistic than previous datasets, incorporating complex visual contexts and implicit multi-step reasoning.
  • Out-of-Distribution (OOD) Benchmarks: The authors evaluate their fine-tuned model on several OOD datasets to assess generalization, including:
  • Seen tools, unseen queries: TextVQA, TallyQA, InfoSeek.
  • Unseen tools: GTA, TEMPLAMA.

Key Insights and Ideas

  • Real-world complexity matters: Synthetic datasets and simplified queries fail to capture the nuances of real-world tool usage, leading to a performance gap in LFMs. ToolVQA aims to bridge this by focusing on realistic scenarios and implicit reasoning.
  • Dynamic example matching is crucial: The LCS-based example matching in ToolEngine significantly improves data quality and reasoning complexity compared to fixed example matching, allowing for better adaptation to diverse scenarios.
  • Fine-tuning is powerful, even for smaller models: Fine-tuning LLaVA-7B on ToolVQA yields substantial improvements, demonstrating that even moderately sized open-source models can achieve impressive tool-use capabilities with the right data.
  • LFMs struggle with multi-step reasoning: The paper highlights that while LFMs can handle single-step reasoning (I → P → Q, A), they face significant challenges with end-to-end multi-step reasoning (I, Q → P → A), indicating a clear gap in their current capabilities.
  • Visual modules need improvement for tool use: For many current LFM architectures, the visual module is less effective at guiding tool usage compared to text-only models, suggesting an area for future architectural development.

Key Results

  • ToolVQA Performance: The LLaVA-7B model fine-tuned on ToolVQA achieved impressive performance on the ToolVQA test set.
  • Out-of-Distribution Generalization: The fine-tuned LLaVA-7B model surpassed the large closed-source model GPT-3.5-Turbo on five OOD datasets (TextVQA, TallyQA, InfoSeek, GTA, TEMPLAMA), demonstrating strong generalization capabilities.
  • LLaVA-7B vs. GPT-3.5-Turbo: The fine-tuned LLaVA-7B model's performance is comparable to or better than GPT-3.5-Turbo on various benchmarks, especially in tool-use tasks.
  • ToolEngine Effectiveness: An ablation study showed that Real-World Example construction and LCS matching significantly improved answer correctness, tool necessity, and reasoning complexity in the generated data.

Limitations and Future Work

  • Argument Prediction and Answer Summarization Bottlenecks: Even with fine-tuning, LFMs still struggle with accurately predicting tool arguments and summarizing complex tool outputs, which requires a deeper understanding of new information.
  • Visual Module Limitations: The visual components of current LFMs are not yet fully optimized for guiding tool usage, leading to VLM+tool settings sometimes underperforming LLM+tool settings.
  • Error Accumulation: Multi-step reasoning is prone to error accumulation, where mistakes in early steps propagate and affect subsequent results.
  • Future Work: The authors suggest that fine-tuning and few-shot in-context learning (ICL) can be complementary, offering potential for further improvement. They also highlight the need for models with higher generalization capabilities in vision-language joint reasoning to address argument prediction and summarization issues. The ToolVQA dataset itself serves as a benchmark for future research in developing more capable tool-using agents.

Summary updated

BibTeX
@inproceedings{yin2025toolvqa,
  author = {Shaofeng Yin and Ting Lei and Yang Liu},
  title = {{ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure