Introduction
Multimodal Large Language Models (MLLMs) demonstrate impressive capabilities in general commonsense tasks but often falter when encountering "long-tail" or domain-specific data. This paper addresses the challenge of reasoning in complex, imaginary ecosystems—specifically the world of Monster Hunter: World—where rules differ from real-world physics. The authors identify three primary hurdles: the domain gap between training data and specialized knowledge, the multi-hop logic required for complex reasoning, and the lack of fine-grained temporal video data in existing knowledge graphs.
The Multi-Agent Retriever Framework
To overcome these challenges, the researchers propose a training-free Multi-agent Retriever framework. This system utilizes an MLLM as an autonomous navigator through a multimodal knowledge graph, employing three specialized agents to handle the search process.
- Topic Selection Agent: Identifies the starting entity (e.g., a specific monster) from the user's query.
- Expansion Agent: Navigates the graph using a Breadth-First Search (BFS) approach to find relevant neighboring nodes.
- Validation Agent: Determines if the retrieved information is sufficient or if further exploration is needed.
A Perceiver MLLM converts video frames into textual descriptions, while a Summarizer synthesizes the final response from the gathered paths.
MH-MMKG and Benchmark
The study introduces MH-MMKG, a high-quality Multimodal Knowledge Graph containing 22 monsters, 265 attack actions, and 20 combat phases, integrated with text, images, and expert-curated video clips. Accompanying this is the MH Benchmark, consisting of 238 expert-crafted queries across six sub-tasks: Individual Information, Action Recognition, Combo Premonition, Condition Awareness, Proc Effect Insight, and Cross-Monster Analysis.
Key Insights
- Closed-World Logic: Video games provide an ideal testbed for reasoning because their internal rules are consistent yet distinct from reality, preventing models from relying on memorized commonsense.
- Online Captioning: Generating video descriptions during retrieval, with the specific question in mind, is more effective than static pre-generation. Contextual awareness helps the model focus on relevant visual details like specific limb movements.
- Search Strategy: Breadth-First Search (BFS) was found to be superior to Depth-First Search (DFS) for this domain, as most critical information resides within a few hops of the primary entity.
Experimental Results
The integration of MH-MMKG leads to substantial performance improvements. In the "Unaided-Online" setting, GPT-4o's accuracy increased significantly from 31.2% to 51.1%.
The results indicate that the primary bottleneck for MLLMs in specialized domains is a lack of domain knowledge rather than visual perception limitations. Even when models were given perfect "human" captions of the videos, they still failed without the Knowledge Graph, proving the necessity of structured domain knowledge for complex reasoning.
Limitations and Future Outlook
Despite the gains, the multi-agent approach is computationally intensive, with high latency due to multiple API calls. Distinguishing between highly similar animations remains a challenge for current visual encoders. Future work will focus on expanding the dataset's scale, exploring hybrid retrieval methods, and incorporating additional modalities like audio to further enhance reasoning in complex environments.
Summary updated