Abstract
Despite the superior performance of Large language models on many NLP tasks, they still face significant limitations in memorizing extensive world knowledge. Recent studies have demonstrated that leveraging the Retrieval-Augmented Generation (RAG) framework, combined with Knowledge Graphs that encapsulate extensive factual data in a structured format, robustly enhances the reasoning capabilities of LLMs. However, deploying such systems in real-world scenarios presents challenges: the continuous evolution of non-stationary environments may lead to performance degradation and user satisfaction requires a careful balance of performance and responsiveness. To address these challenges, we introduce a Multi-objective Multi-Armed Bandit enhanced RAG framework, supported by multiple retrieval methods with diverse capabilities under rich and evolving retrieval contexts in practice. Within this framework, each retrieval method is treated as a distinct "arm". The system utilizes real-time user feedback to adapt to dynamic environments, by selecting the appropriate retrieval method based on input queries and the historical multi-objective performance of each arm. Extensive experiments conducted on two benchmark KGQA datasets demonstrate that our method significantly outperforms baseline methods in non-stationary settings while achieving state-of-the-art performance in stationary environments. Code and data are available at https://github.com/FUTUREEEEEE/Dynamic-RAG.git
Introduction
Retrieval-Augmented Generation (RAG) has become the standard approach for grounding Large Language Models (LLMs) in factual, structured data. When utilizing Knowledge Graphs (KGs), several retrieval paradigms exist, each with distinct trade-offs. Dense retrieval offers high speed but struggles with complex reasoning; KG Query Languages like SPARQL provide precision but are brittle; and LLM Agents are capable of sophisticated reasoning but are often slow and computationally expensive.
Current systems face two primary hurdles in real-world deployment. First is Non-Stationarity (C1): user interests evolve over time, and backend components are frequently updated. Static routing mechanisms fail to adapt to these shifts without costly retraining. Second is the challenge of Multi-Objective Trade-offs (C2): a production-ready system must balance accuracy with responsiveness. Relying on a single retrieval method or a naive ensemble often compromises either quality or latency.
The GGI-MAB Framework
To address these challenges, we propose a Deep Multi-objective Contextual Bandit framework, treating different retrieval backends as distinct "arms." This framework, referred to as GGI-MAB, dynamically selects the best retrieval tool for a specific query.
Dynamic Selection Strategy
The core of the system is a selection loop that maps user queries to the most appropriate retrieval method. We utilize a lightweight DistilBERT model for query encoding, allowing the system to categorize query complexity with minimal overhead. The selection process follows an $\epsilon$-greedy strategy, balancing the exploitation of known high-performing methods with the exploration of alternative "arms" to detect environmental changes or improvements in backend tools.
Multi-Objective Optimization via GGI
To effectively balance competing goals such as accuracy and latency, the system employs the Generalized Gini Index (GGI) as a loss function. GGI serves as a fairness aggregator, preventing the system from over-optimizing for a single metric at the expense of others. This ensures that the selected retrieval method provides a satisfying user experience across all performance dimensions.
Learning from Feedback
The framework adopts an "offline-to-online" learning paradigm. The model is initially pre-trained on historical interaction data and subsequently fine-tuned in real-time using binary user feedback (e.g., thumbs up/down). This continuous learning loop allows the system to self-correct and adapt to shifts in user behavior or backend performance.
Experimental Evaluation
We evaluated the framework on two gold-standard Knowledge Graph Question Answering (KGQA) benchmarks: WebQSP (1-2 hop reasoning) and ComplexWebQuestions (CWQ) (up to 4 hops with multi-entity constraints).
Key Results
- Performance Gains: On the CWQ dataset, GGI-MAB outperformed the strongest baseline by nearly 2% in hit rate and over 2.5% in recall.
- Latency Efficiency: In non-stationary simulations, the proposed method achieved a retrieval delay of 5.88 seconds, a nearly 3x speedup compared to standard ensemble approaches (15.00 seconds), while maintaining superior accuracy.
- Robust Adaptability: When simulating domain shifts, the system maintained a hit rate of 76.35%, significantly outperforming static routers that suffered from performance degradation.
Conclusion
The GGI-MAB framework demonstrates that a "right tool for the job" philosophy, powered by multi-armed bandits, is essential for deploying RAG systems in dynamic environments. By treating user feedback as a primary reward signal and balancing multiple objectives through the Gini Index, the system achieves a robust balance between reasoning power and efficiency. Future work will explore more sophisticated exploration strategies and scaling the framework to accommodate a larger variety of retrieval "arms."
Summary updated