VisTRA: Visual Tool-use Reasoning Analyzer for Small Object Visual Question Answering

Hiroaki Sugiyama, Ko Koga, Toshifumi NishijimaACL 2025

TL;DR

This paper introduces VisTRA, a framework for analyzing how Visual Language Models (VLMs) use tools to answer questions about small objects in high-resolution images. They found that directly accepting tool outputs, rather than verifying them, significantly harms accuracy, and that less capable models often get stuck in unproductive "cyclic loops" of tool use.

What Problem Does This Paper Address?

Visual Question Answering (VQA) systems are getting better, but they struggle with high-resolution images containing small objects. When an entire image is processed at once, small objects become practically invisible. To combat this, researchers have developed tools like object detection and zoom functions that VLMs can use. However, these tools aren't perfect, and VLMs often blindly trust their outputs. This can lead to errors, especially when the tools misidentify objects or fail to find them. The core problem this paper addresses is understanding how VLMs use these tools, particularly how they handle errors and whether they verify tool outputs or just accept them, and how this impacts their overall performance in small object VQA.

Methods

The authors developed VisTRA (Visual Tool-use Reasoning Analyzer), a systematic framework for analyzing VLM tool utilization. VisTRA works by:

  1. Data Collection: Using the V* bench dataset, which is specifically designed for small object VQA.
  2. Systematic Tagging: Creating a fine-grained annotation scheme to categorize each step of a VLM's reasoning process. This includes states for input, tool execution, verification, and acceptance. They use tags to represent actions like "Direct Acceptance" (4:N) or "Verified Acceptance" (4:Y).
  3. Automated Annotation: Developing prompt engineering techniques to automate the VisTRA annotation process, making it scalable across multiple models and datasets.

The framework focuses on analyzing two key behavioral patterns: direct acceptance of tool outputs without verification, and re-planning (switching tools or strategies when an initial attempt fails).

Datasets

  • V* bench: This dataset is used for both training and evaluation. It's highlighted as a benchmark specifically designed for evaluating high-resolution image processing capabilities and small object VQA.

Key Insights and Ideas

  • Direct Acceptance is Bad: The most striking finding is that VLMs that directly accept tool outputs without verification perform worse on VQA tasks. This suggests that a crucial part of robust VQA is a model's ability to critically assess the information provided by its tools.
  • Less Capable Models Get Stuck: Lower-performing VLMs tend to fall into "cyclic loops," where they repeatedly use the same tools without making progress. This indicates a lack of effective re-planning or adaptation when faced with errors.
  • Fine-tuning Reshapes Tool Preferences: Even with minimal training data, fine-tuning can significantly alter a VLM's tool usage patterns. For example, one fine-tuned model reduced its use of segmentation masks, suggesting a shift in its preferred verification strategies.
  • Model Capacity Matters: While behavioral patterns are important, the inherent capacity of a VLM seems to play a more fundamental role in determining its overall VQA performance. More capable models are better at maintaining accuracy even when they directly accept tool outputs.

Key Results

  • Accuracy Drop with Direct Acceptance: Across all models, direct acceptance of tool outputs led to lower VQA accuracy compared to cases where verification occurred. For instance, gpt-40-mini saw an accuracy drop of 14.7 percentage points (from 79.9% to 65.2%) when directly accepting outputs.
  • Model Performance Varies: Gpt-4.5-preview and fine-tuned gpt-4o achieved the highest VQA accuracy (85.3%). Gpt-4.5-preview also had the highest direct acceptance rate (50.0%) and lowest re-planning rate (22.3%). Gpt-40-mini, despite moderate direct acceptance (37.4%), had the highest cyclic loop rate (10.08%) and lowest accuracy (74.4%).
  • Re-planning Effectiveness: Fine-tuned gpt-40 maintained high accuracy (82.5%) during re-planning attempts, while other models showed more degradation.
  • Automated Annotation Accuracy: The VisTRA framework, when automated, achieved good agreement rates with manual annotations, with gpt-4.5-preview and gpt-40 showing the best performance.

Limitations and Future Work

  • Framework Specificity: VisTRA is currently designed for the SKETCHPAD framework and the V* bench dataset. Adapting it to other VQA tasks or tool-use frameworks might require significant prompt engineering.
  • Tool Output Accuracy vs. Model Behavior: The study focused on VLM behavior rather than the accuracy of the tools themselves. Future work should investigate how the interaction between tool output correctness and VLM behavior influences performance.
  • Distinguishing Verification: More sophisticated frameworks are needed to distinguish between genuine direct acceptance and internal verification processes, especially in highly capable models.
  • Scale-Dependent Strategies: Further research is needed to understand if there are scale-dependent patterns in optimal verification strategies that could inform more efficient training.
  • Tool-Specific Confidence: Investigating how models develop tool-specific confidence levels could lead to better training strategies for improving tool use.

Summary updated

BibTeX
@inproceedings{sugiyama2025vistra,
  author = {Hiroaki Sugiyama and Ko Koga and Toshifumi Nishijima},
  title = {{VisTRA: Visual Tool-use Reasoning Analyzer for Small Object Visual Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure