ID-GMLM: Intelligent Decision-Making with Integrated Graph Models and Large Language Models

Zhenhua Meng, Fanshen Meng, Rongheng Lin, Budan WuAAAI 2025 · DOI

TL;DR

This paper introduces ID-GMLM, a novel framework that combines Graph Neural Networks (GNNs) and Large Language Models (LLMs) to enhance intelligent decision-making. By representing decision problems as graphs and leveraging LLMs to interpret human preferences, ID-GMLM achieves superior performance in ranking alternatives compared to existing methods.

What Problem Does This Paper Address?

Intelligent decision support systems are crucial for aiding complex decision-making processes across various domains. Two key subfields are Multi-Criteria Decision Making (MCDM) and Preference Learning (PL), both aiming to help users select, classify, or rank alternatives based on multiple criteria. While MCDM involves active user input to define preferences, PL infers preferences from observed data.

Despite their complementary nature, existing approaches struggle with large datasets and complex relational information between decision factors (alternatives and criteria). Specifically, they face challenges in:

  1. Capturing intricate relationships: Traditional methods often fail to adequately model the complex interactions among decision factors, especially when the number of factors is large.
  2. Parsing dynamic DM preferences: Human preferences can be nuanced, subjective, and change over time. Quantifying these dynamic preferences accurately is difficult for many models, leading to outcomes that may not align with user intuition.

This paper aims to bridge this gap by integrating graph models and LLMs to create a more robust and interpretable intelligent decision-making system.

Methods

The proposed ID-GMLM framework integrates several key components:

  • Graph Representation: Decision problems are reformulated as graphs. Two specific graphs are constructed: an alternative preference graph (Gp) to capture relationships between alternatives and a criterion relationship graph (Ge) for relationships between criteria. Graph Neural Networks (GNNs) are then employed to learn from these graph structures.
  • GNNs-Based Multi-Task Learning:
    • Primary Task (Alternative Preference Network - APN): A GNN processes the alternative preference graph (Gp) to learn representations of alternatives, capturing their interrelationships.
  • Auxiliary Task (Criterion Relationship Network - CRN): Another GNN operates on the criterion relationship graph (Ge) to model interactions among criteria. This multi-task setup helps to capture complex relational dynamics.
  • LLMs-Based Criterion Weights Generation: Large Language Models (LLMs) are used to interpret qualitative descriptions of criteria provided by decision-makers (DMs). These natural language descriptions are transformed into numerical criterion weights, which are crucial inputs for the decision-making process. This addresses the challenge of DM preferences not being explicitly defined.
  • Parameter Tuning Network: A dedicated network uses the LLM-generated criterion weights to dynamically adjust the model's parameters. This allows the model to adapt to evolving DM preferences and fine-tune the influence of different features.
  • Attention Network: An attention mechanism is incorporated to balance the contributions of the primary and auxiliary GNN tasks, ensuring a more effective overall learning process.

The overall optimization objective combines losses from the primary task (alternative ranking), the auxiliary task (criterion relationships), and a regularization term to prevent overfitting.

Datasets

The paper uses six publicly available datasets from the UCI and WEKA repositories. These datasets were chosen because they feature ordered categorical scales and monotonic criterion influences on alternative rankings. Specific datasets mentioned include DBS, BCC, MPG, ESL, LEV, and CEV. No new benchmarks were introduced.

Key Insights and Ideas

  • Synergy of GNNs and LLMs: The core innovation lies in the synergistic combination of GNNs for structural relationship modeling and LLMs for nuanced preference understanding. This pairing addresses limitations of previous approaches that focused on either structure or preference elicitation in isolation.
  • LLMs for Interpretable Preferences: Using LLMs to parse natural language descriptions of criteria into weights is a significant step towards making decision models more interpretable and aligned with human intuition, as DMs often find it easier to express preferences qualitatively.
  • Multi-Task Learning for Comprehensive Modeling: The multi-task learning framework, with the primary task of alternative ranking and the auxiliary task of criterion interaction modeling, allows the model to learn a more holistic representation of the decision problem.
  • Dynamic Adaptation: The parameter tuning network, informed by LLM-generated weights, enables the model to adapt to dynamic DM preferences, a crucial aspect for real-world decision-making.

Key Results

  • Overall Performance: ID-GMLM significantly outperforms several baseline MCDM and PL combination methods across three evaluation metrics (NDCG, AP, and C-index) on all six datasets. For instance, on the BCC dataset, ID-GMLM achieved maximum relative improvements of 20.3% (NDCG), 25.2% (AP), and 24.2% (C-index).
  • Robustness: The model demonstrates stable performance across different datasets, often outperforming baselines by a considerable margin.
  • Component Contributions: Ablation studies show that all components (CRN, Parameter Tuning Network, Attention Network) are crucial for ID-GMLM's performance. The CRN was found to be particularly vital for analyzing criterion relationships.
  • LLM Effectiveness: Using LLM-generated criterion weights (LLM_O) consistently yielded better results than using equal weights (LLM_M) or random weights (LLM_R), highlighting the value of LLM-driven preference interpretation.
  • Parameter Sensitivity: The model's performance is more sensitive to learning rates than regularization weights. The k-nearest neighbors parameter showed optimal performance for k values between 3 and 7.

Limitations and Future Work

  • Limitations: The paper acknowledges that while LLMs improve interpretability, further work is needed to better align DM preferences with LLM outputs. The current approach might still face challenges with extremely complex or ambiguous preference structures.
  • Future Work: The authors suggest future research will focus on improving the alignment between DM preferences and LLM outputs, aiming to further bridge the gap between DM knowledge and data-driven models. This could involve exploring more sophisticated LLM prompting strategies or fine-tuning LLMs specifically for preference elicitation tasks.

Summary updated

BibTeX
@inproceedings{meng2025id,
  author = {Zhenhua Meng and Fanshen Meng and Rongheng Lin and Budan Wu},
  title = {{ID-GMLM: Intelligent Decision-Making with Integrated Graph Models and Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34142},
}
Figure