Abstract
Multi-Criteria Decision Making (MCDM) is a critical component of rational choice in complex systems, yet it remains bottlenecked by the need for manual expert intervention. This paper presents a general framework that leverages Large Language Models (LLMs) to automate MCDM at a human-expert level. By employing Low-Rank Adaptation (LoRA) fine-tuning on a modest dataset of 500 samples, we demonstrate that open-source LLMs can achieve approximately 95% accuracy, significantly outperforming base models and sophisticated prompting techniques. Our results suggest that fine-tuning is the key to bridging the expertise gap, enabling scalable and consistent decision-making across diverse domains such as supplier evaluation and environmental monitoring.
Introduction
MCDM is the process of evaluating multiple conflicting criteria—such as cost, quality, and speed—to make rational choices. Standard methods like the Analytic Hierarchy Process (AHP) or Fuzzy Comprehensive Evaluation (FCE) are powerful but suffer from scalability issues and a heavy reliance on human experts to manually construct judgment matrices. While LLMs are excellent at processing text, their ability to handle the specific logical rigors of MCDM has not been systematically optimized until now.
Methodology
The authors propose a general framework that evaluates and enhances LLMs for MCDM tasks through three main approaches: prompt engineering, LoRA fine-tuning, and hybrid ground truth establishment.
LoRA Fine-tuning
The core contribution involves using Low-Rank Adaptation (LoRA) to fine-tune open-source models (Llama3-8B, Qwen2-7B, and ChatGLM4-9B) on just 500 domain-specific examples. This allows the models to learn the underlying evaluation logic of human experts.
Hybrid Ground Truth
To evaluate the LLMs, the authors established a "ground truth" using a hybrid AHP-FCE method. This traditional mathematical approach provides the benchmark for what a "correct" expert decision looks like.
Datasets
The framework was validated across three distinct MCDM applications:
- Supplier Evaluation: Assessing reliability based on delivery timeliness, product quality, and supply capability.
- Customer Satisfaction: A synthetic dataset used to analyze and predict customer behavior and satisfaction levels.
- Air Quality Evaluation: A comprehensive dataset involving pollutants and population densities to assign ratings from "Good" to "Hazardous."
Key Results and Insights
Standard commercial models are surprisingly mediocre at MCDM out of the box, typically hovering around 60% accuracy. While Chain-of-Thought (CoT) and few-shot prompting improve performance to roughly 70-75%, they cannot bridge the final gap to expert-level performance.
The Power of Fine-tuning
After LoRA fine-tuning, the performance gap between different model architectures virtually disappears. All fine-tuned models reached ~95% accuracy, suggesting that the method of adaptation is more important than the base model's size for these specific tasks.
LoRA-tuned models achieved F1 scores between 0.91 and 0.98 across all datasets, a massive jump from the ~0.55-0.65 range of base models. The fine-tuned models showed remarkably stable performance, with confusion matrices showing very few "off-diagonal" errors, meaning they rarely confuse extreme ratings like "Good" and "Poor."
Conclusion
This research demonstrates that LLMs can serve as effective MCDM experts when properly fine-tuned. Future work will focus on reducing data dependency through synthetic data generation and testing the framework's robustness against "messy" real-world data with missing or ambiguous values.
Summary updated