Abstract
Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have transformed information retrieval, yet they often struggle in high-stakes decision-making environments that require transparent, quantifiable reasoning. This paper introduces the Retrieval Augmented Decision-Making (RAD) framework, which integrates Multi-Criteria Decision Making (MCDM) with LLMs. By automatically constructing weighted, hierarchical models from complex documents, RAD provides a structured alternative to the "flat" context of standard RAG. Our evaluation shows that RAD achieves significantly higher win rates in evidence-based reasoning and maintains superior structural accuracy compared to existing baselines.
Introduction
In professional sectors such as industrial planning and regulatory compliance, decision-makers require more than just summarized text; they need a clear reasoning architecture. Standard RAG systems often fail here because they lack the ability to assign quantitative weights to different factors or provide a traceable chain of logic. The RAD framework addresses this by moving beyond simple text generation into the realm of structured decision analysis, ensuring that criteria like cost, safety, and efficiency are balanced transparently.
The RAD Framework
The RAD framework employs a sophisticated four-stage pipeline to transform unstructured data into actionable intelligence:
- Text Processing: Unlike arbitrary word-count chunking, RAD uses Content Hierarchies Extraction. By parsing a document's table of contents, it preserves semantic integrity, ensuring sections remain cohesive units.
- Criteria Extraction: LLMs identify key decision criteria and their underlying logical relationships, focusing on factors that directly influence outcomes rather than simple entity recognition.
- Hierarchical Modeling: This stage utilizes Interpretive Structural Modeling (ISM) to organize criteria into logical levels. A multi-agent voting mechanism involving five specialized LLM "experts" ranks these criteria, which are then converted into mathematical weights using the Analytic Hierarchy Process (AHP).
- Decision-making: The system scores candidate options against the weighted hierarchy, performs consistency checks, and generates a structured report detailing final scores and reasoning.
Experimental Evaluation
RAD was evaluated using several distinct datasets, including prompt engineering materials (R1, ~17.8k tokens) and official WHO COVID-19 therapeutic guidelines (R2, ~186k tokens). Additionally, documents from the Federal Register (EVID & ESPD) were used to test the rationality of the generated hierarchical models.
Results and Discussion
The experimental results highlight the superiority of structured frameworks over raw LLM generation. In head-to-head comparisons, RAD achieved a 70% win rate over non-hierarchical methods in evidence-based reasoning. On the complex COVID-19 dataset (R2), RAD's directory-based segmentation outperformed standard fixed-word chunking with a 63% win rate in matching specific user needs.
Furthermore, RAD demonstrated exceptional structural accuracy, identifying correct decision criteria with 94-95% accuracy and parent-child relationships within the hierarchy with 92-96% accuracy. When compared to GraphRAG, RAD proved more effective for specific decision tasks, as its hierarchical focus better respected user-defined constraints than GraphRAG's broader global search.
Conclusion and Future Work
RAD demonstrates that forcing LLMs to operate within classic decision-analysis frameworks like AHP and ISM significantly improves performance and transparency. While the system currently excels with text, future work will focus on integrating multimodal data—such as charts and flow diagrams—into the hierarchical models. Additionally, while the multi-agent approach mitigates subjectivity in weighting, further refinement is needed to better capture nuanced human preferences in complex scenarios.
Summary updated