Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering

Zhongjian Hu, Peng Yang, Bing Li, Zhenqi Wang

TL;DR

This paper introduces a novel multi-agent framework for knowledge-based Visual Question Answering (VQA) that mimics human teamwork. By leveraging Large Language Models (LLMs) as agents with varying tool access and a voting mechanism, the approach significantly improves VQA performance on challenging datasets.

What Problem Does This Paper Address?

Knowledge-based Visual Question Answering (VQA) requires not only understanding visual content but also accessing and integrating external knowledge. While Large Language Models (LLMs) have shown promise in this area, existing methods struggle with two key limitations: the inability to autonomously use external tools and the lack of collaborative capabilities. Humans, when faced with complex questions, often know when to consult external resources (like search engines) and can collaborate with others to arrive at better answers. This paper aims to bridge this gap by creating an LLM-based system that can autonomously utilize tools and work in a team-like structure to solve knowledge-based VQA tasks.

Methods

The core of this work is the Multi-Agents Voting framework (MAVL). This framework comprises three distinct LLM-based agents, designed to simulate different levels of expertise within a team: Junior, Senior, and Manager. Each agent is equipped with a planner and a set of tools.

  • Planner: This component, itself based on an LLM, allows each agent to autonomously plan a sequence of actions to address a given VQA task. The planner decides which tools to invoke based on the agent's capabilities.
  • Tools: Three types of tools are defined:
  • Candidate Generator: Uses a vanilla VQA model to generate a list of potential answers.
  • KBs Retriever: Fetches relevant knowledge from external Knowledge Bases (KBs).
  • LLM Generator: Leverages an LLM to generate new knowledge relevant to the query.
  • Agent Capabilities: The agents have tiered access to these tools:
  • Junior Agent: Can only use Tool 1 (Candidate Generator).
  • Senior Agent: Can use Tool 1 (Candidate Generator) and Tool 2 (KBs Retriever).
  • Manager Agent: Can use all three tools (Candidate Generator, KBs Retriever, and LLM Generator).
  • Voting Mechanism: After each agent generates its answer, a voting process takes place. The agents' answers are weighted based on their level (Manager > Senior > Junior), and the final answer is determined by the majority vote. This collaborative approach aims to leverage the diverse perspectives and capabilities of the agents.

Datasets

The proposed framework was evaluated on two widely used knowledge-based VQA datasets:

  • OK-VQA: Contains approximately 9K training and 5K test samples, with each sample annotated with 10 answers.
  • A-OKVQA: Features a larger dataset with about 17K training, 1K validation, and 7K test samples.

Key Insights and Ideas

  • Mimicking Human Collaboration: The most significant insight is the successful application of a multi-agent, team-based approach to knowledge-based VQA, drawing inspiration from how humans collaborate.
  • Tiered Agent Capabilities: Differentiating agent capabilities based on tool access (Junior, Senior, Manager) allows for a more nuanced and effective problem-solving process.
  • Autonomous Tool Invocation: The planner component enables agents to decide and execute tool usage autonomously, a crucial step beyond static pipelines.
  • Complementary Knowledge Sources: Combining candidate generation, KB retrieval, and LLM-generated knowledge provides a robust way to gather necessary information.
  • Weighted Voting for Robustness: The weighted voting mechanism ensures that the more capable agents have a greater influence on the final decision, leading to more accurate outcomes.

Key Results

The proposed multi-agent framework achieved state-of-the-art results on both OK-VQA and A-OKVQA datasets. Specifically, it outperformed existing baselines by 2.2% on OK-VQA and 1.0% on A-OKVQA. Ablation studies confirmed the importance of each component: removing multi-agents, tools, or planners led to performance degradation, highlighting the synergistic effect of the proposed framework.

Limitations and Future Work

The paper acknowledges that while the framework is based on open LLMs and can run on a single GPU (LLaMA2 7B), the performance is still influenced by the number of in-context examples and candidate answers. The authors suggest that increasing these parameters generally improves performance up to a certain point, after which it plateaus. Future work could explore more sophisticated voting strategies, dynamic agent role assignment, and extending this multi-agent concept to other complex AI tasks beyond VQA.

Summary updated

Abstract

Large Language Models (LLMs) have achieved impressive results in knowledge-based Visual Question Answering (VQA). However existing methods still have challenges: the inability to use external tools autonomously, and the inability to work in teams. Humans tend to know whether they need to use external tools when they encounter a new question, e.g., they tend to be able to give a direct answer to a familiar question, whereas they tend to use tools such as search engines when they encounter an unfamiliar question. In addition, humans also tend to collaborate and discuss with others to get better answers. Inspired by this, we propose the multi-agent voting framework. We design three LLM-based agents that simulate different levels of staff in a team, and assign the available tools according to the levels. Each agent provides the corresponding answer, and finally all the answers provided by the agents are voted to get the final answer. Experiments on OK-VQA and A-OKVQA show that our approach outperforms other baselines by 2.2 and 1.0, respectively.
BibTeX
@misc{hu2024multi,
  author = {Zhongjian Hu and Peng Yang and Bing Li and Zhenqi Wang},
  title = {{Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering}},
  year = {2024},
  eprint = {2412.18351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.18351},
}
Figure