One for All: A General Framework of LLMs-based Multi-Criteria Decision Making on Human Expert Level

Hui Wang, Fafa Zhang, Chaoxu MuarXiv 2025 · DOI · arXiv

Abstract

Multi-Criteria Decision Making (MCDM) is a critical component of rational choice in complex systems, yet it remains bottlenecked by the need for manual expert intervention. This paper presents a general framework that leverages Large Language Models (LLMs) to automate MCDM at a human-expert level. By employing Low-Rank Adaptation (LoRA) fine-tuning on a modest dataset of 500 samples, we demonstrate that open-source LLMs can achieve approximately 95% accuracy, significantly outperforming base models and sophisticated prompting techniques. Our results suggest that fine-tuning is the key to bridging the expertise gap, enabling scalable and consistent decision-making across diverse domains such as supplier evaluation and environmental monitoring.

Introduction

MCDM is the process of evaluating multiple conflicting criteria—such as cost, quality, and speed—to make rational choices. Standard methods like the Analytic Hierarchy Process (AHP) or Fuzzy Comprehensive Evaluation (FCE) are powerful but suffer from scalability issues and a heavy reliance on human experts to manually construct judgment matrices. While LLMs are excellent at processing text, their ability to handle the specific logical rigors of MCDM has not been systematically optimized until now.

Methodology

The authors propose a general framework that evaluates and enhances LLMs for MCDM tasks through three main approaches: prompt engineering, LoRA fine-tuning, and hybrid ground truth establishment.

The proposed general framework for LLM-based MCDM, integrating prompt design, LoRA fine-tuning, and traditional AHP-FCE evaluation processes.

LoRA Fine-tuning

The core contribution involves using Low-Rank Adaptation (LoRA) to fine-tune open-source models (Llama3-8B, Qwen2-7B, and ChatGLM4-9B) on just 500 domain-specific examples. This allows the models to learn the underlying evaluation logic of human experts.

Hybrid Ground Truth

To evaluate the LLMs, the authors established a "ground truth" using a hybrid AHP-FCE method. This traditional mathematical approach provides the benchmark for what a "correct" expert decision looks like.

Datasets

The framework was validated across three distinct MCDM applications:

  1. Supplier Evaluation: Assessing reliability based on delivery timeliness, product quality, and supply capability.
  2. Customer Satisfaction: A synthetic dataset used to analyze and predict customer behavior and satisfaction levels.
  3. Air Quality Evaluation: A comprehensive dataset involving pollutants and population densities to assign ratings from "Good" to "Hazardous."

Key Results and Insights

Standard commercial models are surprisingly mediocre at MCDM out of the box, typically hovering around 60% accuracy. While Chain-of-Thought (CoT) and few-shot prompting improve performance to roughly 70-75%, they cannot bridge the final gap to expert-level performance.

The Power of Fine-tuning

After LoRA fine-tuning, the performance gap between different model architectures virtually disappears. All fine-tuned models reached ~95% accuracy, suggesting that the method of adaptation is more important than the base model's size for these specific tasks.

Confusion matrix for the LoRA-tuned Llama3-8B model on the Customer Satisfaction task, showing high diagonal concentration and minimal errors.

LoRA-tuned models achieved F1 scores between 0.91 and 0.98 across all datasets, a massive jump from the ~0.55-0.65 range of base models. The fine-tuned models showed remarkably stable performance, with confusion matrices showing very few "off-diagonal" errors, meaning they rarely confuse extreme ratings like "Good" and "Poor."

Conclusion

This research demonstrates that LLMs can serve as effective MCDM experts when properly fine-tuned. Future work will focus on reducing data dependency through synthetic data generation and testing the framework's robustness against "messy" real-world data with missing or ambiguous values.

Summary updated

Abstract

Multi-Criteria Decision Making~(MCDM) is widely applied in various fields, using quantitative and qualitative analyses of multiple levels and attributes to support decision makers in making scientific and rational decisions in complex scenarios. However, traditional MCDM methods face bottlenecks in high-dimensional problems. Given the fact that Large Language Models~(LLMs) achieve impressive performance in various complex tasks, but limited work evaluates LLMs in specific MCDM problems with the help of human domain experts, we further explore the capability of LLMs by proposing an LLM-based evaluation framework to automatically deal with general complex MCDM problems. Within the framework, we assess the performance of various typical open-source models, as well as commercial models such as Claude and ChatGPT, on 3 important applications, these models can only achieve around 60\% accuracy rate compared to the evaluation ground truth. Upon incorporation of Chain-of-Thought or few-shot prompting, the accuracy rates rise to around 70\%, and highly depend on the model. In order to further improve the performance, a LoRA-based fine-tuning technique is employed. The experimental results show that the accuracy rates for different applications improve significantly to around 95\%, and the performance difference is trivial between different models, indicating that LoRA-based fine-tuned LLMs exhibit significant and stable advantages in addressing MCDM tasks and can provide human-expert-level solutions to a wide range of MCDM challenges.
BibTeX
@misc{wang2025one,
  author = {Hui Wang and Fafa Zhang and Chaoxu Mu},
  title = {{One for All: A General Framework of LLMs-based Multi-Criteria Decision Making on Human Expert Level}},
  year = {2025},
  eprint = {2502.15778},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.15778},
}
Figure