TL;DR
This paper introduces a novel multi-agent framework for knowledge-based Visual Question Answering (VQA) that mimics human teamwork. By leveraging Large Language Models (LLMs) as agents with varying tool access and a voting mechanism, the approach significantly improves VQA performance on challenging datasets.
What Problem Does This Paper Address?
Knowledge-based Visual Question Answering (VQA) requires not only understanding visual content but also accessing and integrating external knowledge. While Large Language Models (LLMs) have shown promise in this area, existing methods struggle with two key limitations: the inability to autonomously use external tools and the lack of collaborative capabilities. Humans, when faced with complex questions, often know when to consult external resources (like search engines) and can collaborate with others to arrive at better answers. This paper aims to bridge this gap by creating an LLM-based system that can autonomously utilize tools and work in a team-like structure to solve knowledge-based VQA tasks.
Methods
The core of this work is the Multi-Agents Voting framework (MAVL). This framework comprises three distinct LLM-based agents, designed to simulate different levels of expertise within a team: Junior, Senior, and Manager. Each agent is equipped with a planner and a set of tools.
- Planner: This component, itself based on an LLM, allows each agent to autonomously plan a sequence of actions to address a given VQA task. The planner decides which tools to invoke based on the agent's capabilities.
- Tools: Three types of tools are defined:
- Candidate Generator: Uses a vanilla VQA model to generate a list of potential answers.
- KBs Retriever: Fetches relevant knowledge from external Knowledge Bases (KBs).
- LLM Generator: Leverages an LLM to generate new knowledge relevant to the query.
- Agent Capabilities: The agents have tiered access to these tools:
- Junior Agent: Can only use Tool 1 (Candidate Generator).
- Senior Agent: Can use Tool 1 (Candidate Generator) and Tool 2 (KBs Retriever).
- Manager Agent: Can use all three tools (Candidate Generator, KBs Retriever, and LLM Generator).
- Voting Mechanism: After each agent generates its answer, a voting process takes place. The agents' answers are weighted based on their level (Manager > Senior > Junior), and the final answer is determined by the majority vote. This collaborative approach aims to leverage the diverse perspectives and capabilities of the agents.
Datasets
The proposed framework was evaluated on two widely used knowledge-based VQA datasets:
- OK-VQA: Contains approximately 9K training and 5K test samples, with each sample annotated with 10 answers.
- A-OKVQA: Features a larger dataset with about 17K training, 1K validation, and 7K test samples.
Key Insights and Ideas
- Mimicking Human Collaboration: The most significant insight is the successful application of a multi-agent, team-based approach to knowledge-based VQA, drawing inspiration from how humans collaborate.
- Tiered Agent Capabilities: Differentiating agent capabilities based on tool access (Junior, Senior, Manager) allows for a more nuanced and effective problem-solving process.
- Autonomous Tool Invocation: The planner component enables agents to decide and execute tool usage autonomously, a crucial step beyond static pipelines.
- Complementary Knowledge Sources: Combining candidate generation, KB retrieval, and LLM-generated knowledge provides a robust way to gather necessary information.
- Weighted Voting for Robustness: The weighted voting mechanism ensures that the more capable agents have a greater influence on the final decision, leading to more accurate outcomes.
Key Results
The proposed multi-agent framework achieved state-of-the-art results on both OK-VQA and A-OKVQA datasets. Specifically, it outperformed existing baselines by 2.2% on OK-VQA and 1.0% on A-OKVQA. Ablation studies confirmed the importance of each component: removing multi-agents, tools, or planners led to performance degradation, highlighting the synergistic effect of the proposed framework.
Limitations and Future Work
The paper acknowledges that while the framework is based on open LLMs and can run on a single GPU (LLaMA2 7B), the performance is still influenced by the number of in-context examples and candidate answers. The authors suggest that increasing these parameters generally improves performance up to a certain point, after which it plateaus. Future work could explore more sophisticated voting strategies, dynamic agent role assignment, and extending this multi-agent concept to other complex AI tasks beyond VQA.
Summary updated