Abstract
Large Language Models (LLMs) often struggle with hallucinations in knowledge-intensive domains due to their reliance on static training data. Retrieval-Augmented Generation (RAG) addresses this by incorporating external knowledge. Textual graphs, which represent entities as nodes and relationships as edges, offer structured and semantically rich information for reasoning. However, existing graph-based RAG systems typically rely on a single retriever, failing to capture diverse query intents and often retrieving irrelevant noise. We present MIXRAG, a Mixture-of-Experts Graph-RAG framework that utilizes multiple specialized retrievers and a dynamic routing controller. By adaptively fusing experts focused on entities, relations, and subgraph topology, and employing a query-aware GraphEncoder to filter noise, MIXRAG achieves state-of-the-art performance across diverse graph-based question-answering tasks.
Introduction
While LLMs have revolutionized natural language processing, their performance on complex, fact-based queries is limited by the "knowledge cutoff" of their training sets. RAG systems mitigate this by fetching relevant documents during inference. Textual graphs provide a superior alternative to flat text for many tasks, as they explicitly model the connections between concepts. Despite their potential, retrieving the precise information needed from a large graph remains a significant challenge.
The Challenge: Noise and Diverse Intents
Existing graph-based RAG systems face two primary hurdles. First, a single retrieval strategy is often insufficient for the variety of questions users ask. Some queries require identifying a specific node, while others necessitate tracing multi-hop paths. Second, retrieved subgraphs are frequently "noisy," containing nodes that are structurally adjacent to the target information but semantically irrelevant to the specific query.
The MIXRAG Architecture
MIXRAG introduces a modular approach to graph retrieval, treating it as a multi-specialist task. The framework consists of three main stages: Question Encoding, Mixture-of-Experts (MoE) Retrieval, and Answer Generation.
1. Mixture-of-Experts Retriever
Instead of a monolithic retriever, MIXRAG employs three specialized experts:
- Entity Expert: Optimized for pinpointing specific nodes and their immediate attributes.
- Relation Expert: Focused on the semantic links and predicates between entities.
- Subgraph Expert: Designed to extract interconnected neighborhoods for complex, multi-hop reasoning.
A learned routing controller analyzes the input query embedding to dynamically weight the contribution of each expert, ensuring the retrieval strategy matches the query's intent.
2. Query-Aware GraphEncoder
To address the noise problem, MIXRAG utilizes a Graph Neural Network (GNN) encoder that is conditioned on the query. During the message-passing phase, the encoder calculates the relevance of each node and edge relative to the question. This allows the system to highlight relevant reasoning paths and down-weight distractors before the information is passed to the LLM.
3. Knowledge Fusion and Answer Generation
The retrieved and filtered knowledge is fused into a "soft prompt"—a continuous vector representation. This representation, along with the original query, is fed into a frozen LLM to generate the final grounded answer.
Evaluation
We evaluated MIXRAG on the GraphQA benchmark, which includes three diverse datasets:
- ExplaGraphs: Commonsense reasoning regarding claim support/refutation.
- SceneGraphs: Spatial reasoning within visual environments.
- WebQSP: Complex natural language questions over large-scale knowledge bases (Freebase).
Key Results
MIXRAG consistently outperformed existing state-of-the-art (SOTA) baselines, such as G-Retriever. Ablation studies confirmed that the synergy of all three experts provides the most robust performance across different query types.
| Dataset | Metric | MIXRAG | Previous SOTA |
|---|---|---|---|
| ExplaGraphs | Accuracy | 0.8863 | 0.8705 |
| SceneGraphs | Accuracy | 0.8712 | 0.8683 |
| WebQSP | Hit@1 | 75.31 | 73.79 |
Conclusion
MIXRAG demonstrates that a "one-size-fits-all" approach to graph retrieval is suboptimal. By leveraging a Mixture-of-Experts architecture and query-aware noise filtering, MIXRAG provides a more flexible and accurate framework for textual graph understanding. Future work will focus on scaling this approach to even larger graph structures and exploring automated discovery of new expert types.
Summary updated