MMGraph-Agent: Agentic Multimodal RAG via Cache-Inspired Multimodal Knowledge HyperGraphs

Haoran Luo, Ziyue Zhu, Shangyang Wu, Linhao Luo, Yikai Guo, Qika Lin, Fangzhi Xu, Jiapu Wang, Xiaobao Wu, Yifan Zhu, Luu Anh TuanarXiv 2026 · arXiv

TL;DR

MMGraph-Agent is a unified multimodal Retrieval-Augmented Generation (RAG) framework that slashes the cost and complexity of building agentic AI systems. By replacing expensive LLM-based graph construction with a "zero-API-cost" hypergraph and implementing a two-tier cache memory system, it achieves an average 11.6% F1 improvement across 23 datasets while reducing context token usage by up to 67%.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) is evolving. We are moving past simple text-chunking toward "Agentic Multimodal RAG," where AI agents decide what to retrieve from a mix of text, images, and tables across both private offline databases and the live web.

However, three major bottlenecks persist:

  1. High Construction Costs: Building a Knowledge Graph (KG) typically requires calling expensive LLM APIs (like GPT-4) to extract entities and relations. For large datasets, this is prohibitively expensive and slow.
  2. Context Overflow: In multi-turn reasoning, an agent retrieves more evidence at every step. This quickly fills up the model's context window, leading to performance degradation or outright failure.
  3. The Offline-Online Split: Most systems are optimized for either private "offline" data or "online" web search. Training a single agent that can seamlessly navigate both remains a challenge.
Figure 1: Target challenges and MMGraph-Agent.

Methods

The authors introduce MMGraph-Agent, which tackles these issues through three core modules:

1. Zero-API-Cost Multimodal Hypergraph (MMKHG)

Instead of using an LLM to build the graph, the authors use deterministic local tools: spaCy for Named Entity Recognition (NER) and CLIP for multimodal embeddings. They organize data into hypergraphs, where a single "hyperedge" connects all entities found within a sentence, preserving the original context better than traditional pairwise (A $\to$ B) relations. This process runs entirely on local CPUs/GPUs, making it essentially free and "ultra-fast."

2. Cache-Inspired Two-Tier Memory

To solve the context window problem, the system mimics a CPU's memory hierarchy:

  • L2 (Backend): The full, persistent multimodal hypergraph.
  • L1 (Trajectory Cache): A per-query cache that stores evidence retrieved during the current session. Instead of feeding raw retrieved chunks into the LLM, the agent receives a concise digest (summary) of the L1 cache. This keeps the prompt length bounded and stable, even over long reasoning chains.

3. Unified Agentic Policy

The agent is trained using Group Relative Policy Optimization (GRPO). Crucially, it is trained only on offline data but uses a "source-agnostic" interface. This allows the same trained policy to be deployed for online web search at inference time without any additional fine-tuning.

Figure 2: Overview of MMGraph-Agent: a multimodal RAG framework built on cache-inspired knowledge hypergraphs supporting unified joint retrieval with reduced cost and context overhead.

Datasets

The researchers evaluated MMGraph-Agent on a massive suite of 23 multimodal QA datasets, categorized by retrieval regime:

  • Knowledge-Only (9): Including DocBench, MMLongBench-Doc, and HotpotQA.
  • WebSearch-Only (5): Including InfoSeek and MMSearch.
  • Knowledge + WebSearch (9): Including MedXpertQA, FinMME, and GeoGrid-Bench.

Notably, 8 of these datasets were held out as "Out-of-Distribution" (OOD) to test how well the agent generalizes to unseen data types.

Key Insights and Ideas

  • Hyperedges are superior for RAG: The paper demonstrates that $n$-ary hyperedges (linking multiple entities at once) are more effective for cross-modal reasoning than standard pairwise edges. In a head-to-head comparison, the hypergraph representation won on 23/23 datasets.
  • Deterministic extraction is "Good Enough": You don't need a $20/month LLM subscription to build a high-quality knowledge base. Local NER and CLIP models provide comparable coverage for a fraction of the cost.
  • The "Digest" is the secret to long-context stability: By summarizing retrieved evidence into a fixed-size digest, the system avoids the "lost in the middle" phenomenon and context overflow.
  • Zero-Shot Online Transfer: The most surprising finding is that an agent trained to query an offline hypergraph can effectively query the web via SerpAPI without ever seeing a web page during training.

Key Results

  • Performance: MMGraph-Agent-3B outperformed all baselines, including those using much larger models like GPT-4. It achieved a massive +44.93 F1 gain on the FinMME financial dataset and +23.06 on ViDoSeek.
  • Cost-Efficiency: As shown in the Pareto frontier (Figure 5), MMGraph-Agent achieves the highest accuracy at zero API construction cost.
  • Context Economy: The L1 cache system saved 67% of tokens compared to a no-cache configuration, keeping the context window well within the 8K limit even after 5 rounds of retrieval.
Cost-performance Pareto frontier showing MMGraph-Agent's efficiency.

Limitations and Future Work

While impressive, the authors acknowledge several areas for growth:

  • Construction Overhead: While "zero-API-cost," there is still a one-time local compute cost to build the hypergraph.
  • Web Dependencies: Online retrieval still depends on external search APIs (like SerpAPI), which incur their own costs at inference time.
  • Scaling: The current experiments focused on a 3B parameter backbone. Scaling to larger models and broader modalities (audio, video) is a natural next step.
  • Learned Constants: Currently, memory constants (like cache size) are manually set. Future versions could learn these dynamically via the RL loop.

Summary updated

Abstract

Driven by increasingly complex real-world applications, Retrieval-Augmented Generation (RAG) has evolved from textual pipelines to multimodal settings, comprising two complementary dimensions: knowledge organization and retrieval enhancement. However, existing methods still face challenges of high construction cost, context window limitations, and the lack of unified training across offline and online knowledge sources. To address these challenges, we propose MMGraph-Agent, a unified multimodal RAG framework based on cache-inspired multimodal knowledge hypergraphs. MMGraph-Agent enables extremely lightweight hypergraph construction with zero API cost and dynamic memory-based retrieval that alleviates long-context issues. Experiments across offline, online, and joint retrieval settings demonstrate consistent improvements in performance, efficiency, and architectural unification. Our software and data are publicly available at https://github.com/LHRLAB/MMGraph-Agent.
BibTeX
@misc{luo2026mmgraph,
  author = {Haoran Luo and Ziyue Zhu and Shangyang Wu and Linhao Luo and Yikai Guo and Qika Lin and Fangzhi Xu and Jiapu Wang and Xiaobao Wu and Yifan Zhu and Luu Anh Tuan},
  title = {{MMGraph-Agent: Agentic Multimodal RAG via Cache-Inspired Multimodal Knowledge HyperGraphs}},
  year = {2026},
  eprint = {2611.10657},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10657},
}
Figure