Retrieval Augmented Decision-Making: A Requirements-Driven, Multi-Criteria Framework for Structured Decision Support

Hongjia Wu, Hongxin Zhang, Wei Chen, Jiazhi XiaarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have transformed information retrieval, yet they often struggle in high-stakes decision-making environments that require transparent, quantifiable reasoning. This paper introduces the Retrieval Augmented Decision-Making (RAD) framework, which integrates Multi-Criteria Decision Making (MCDM) with LLMs. By automatically constructing weighted, hierarchical models from complex documents, RAD provides a structured alternative to the "flat" context of standard RAG. Our evaluation shows that RAD achieves significantly higher win rates in evidence-based reasoning and maintains superior structural accuracy compared to existing baselines.

Introduction

In professional sectors such as industrial planning and regulatory compliance, decision-makers require more than just summarized text; they need a clear reasoning architecture. Standard RAG systems often fail here because they lack the ability to assign quantitative weights to different factors or provide a traceable chain of logic. The RAD framework addresses this by moving beyond simple text generation into the realm of structured decision analysis, ensuring that criteria like cost, safety, and efficiency are balanced transparently.

The RAD Framework

The RAD framework employs a sophisticated four-stage pipeline to transform unstructured data into actionable intelligence:

  1. Text Processing: Unlike arbitrary word-count chunking, RAD uses Content Hierarchies Extraction. By parsing a document's table of contents, it preserves semantic integrity, ensuring sections remain cohesive units.
  2. Criteria Extraction: LLMs identify key decision criteria and their underlying logical relationships, focusing on factors that directly influence outcomes rather than simple entity recognition.
  3. Hierarchical Modeling: This stage utilizes Interpretive Structural Modeling (ISM) to organize criteria into logical levels. A multi-agent voting mechanism involving five specialized LLM "experts" ranks these criteria, which are then converted into mathematical weights using the Analytic Hierarchy Process (AHP).
  4. Decision-making: The system scores candidate options against the weighted hierarchy, performs consistency checks, and generates a structured report detailing final scores and reasoning.
The four-stage pipeline of the RAD framework, illustrating the flow from text processing to final decision-making.

Experimental Evaluation

RAD was evaluated using several distinct datasets, including prompt engineering materials (R1, ~17.8k tokens) and official WHO COVID-19 therapeutic guidelines (R2, ~186k tokens). Additionally, documents from the Federal Register (EVID & ESPD) were used to test the rationality of the generated hierarchical models.

Results and Discussion

The experimental results highlight the superiority of structured frameworks over raw LLM generation. In head-to-head comparisons, RAD achieved a 70% win rate over non-hierarchical methods in evidence-based reasoning. On the complex COVID-19 dataset (R2), RAD's directory-based segmentation outperformed standard fixed-word chunking with a 63% win rate in matching specific user needs.

Comprehensive head-to-head win rates for RAD against various baselines across different criteria (C1, C2, C3) and datasets (R1, R2).

Furthermore, RAD demonstrated exceptional structural accuracy, identifying correct decision criteria with 94-95% accuracy and parent-child relationships within the hierarchy with 92-96% accuracy. When compared to GraphRAG, RAD proved more effective for specific decision tasks, as its hierarchical focus better respected user-defined constraints than GraphRAG's broader global search.

Conclusion and Future Work

RAD demonstrates that forcing LLMs to operate within classic decision-analysis frameworks like AHP and ISM significantly improves performance and transparency. While the system currently excels with text, future work will focus on integrating multimodal data—such as charts and flow diagrams—into the hierarchical models. Additionally, while the multi-agent approach mitigates subjectivity in weighting, further refinement is needed to better capture nuanced human preferences in complex scenarios.

Summary updated

Abstract

Various industries have produced a large number of documents such as industrial plans, technical guidelines, and regulations that are structurally complex and content-wise fragmented. This poses significant challenges for experts and decision-makers in terms of retrieval and understanding. Although existing LLM-based Retrieval-Augmented Generation methods can provide context-related suggestions, they lack quantitative weighting and traceable reasoning paths, making it difficult to offer multi-level and transparent decision support. To address this issue, this paper proposes the RAD method, which integrates Multi-Criteria Decision Making with the semantic understanding capabilities of LLMs. The method automatically extracts key criteria from industry documents, builds a weighted hierarchical decision model, and generates structured reports under model guidance. The RAD framework introduces explicit weight assignment and reasoning chains in decision generation to ensure accuracy, completeness, and traceability. Experiments show that in various decision-making tasks, the decision reports generated by RAD significantly outperform existing methods in terms of detail, rationality, and structure, demonstrating its application value and potential in complex decision support scenarios.
BibTeX
@misc{wu2025retrieval,
  author = {Hongjia Wu and Hongxin Zhang and Wei Chen and Jiazhi Xia},
  title = {{Retrieval Augmented Decision-Making: A Requirements-Driven, Multi-Criteria Framework for Structured Decision Support}},
  year = {2025},
  eprint = {2505.18483},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.18483},
}
Figure