Abstract
Understanding complex social-ecological systems requires capturing the mental models of diverse stakeholders, often represented as Fuzzy Cognitive Maps (FCMs). However, manual extraction of these causal graphs from text is labor-intensive, and traditional evaluation metrics fail to account for semantic nuances. This paper introduces a framework for automated FCM extraction using Large Language Models (LLMs) and proposes "soft" evaluation measures that reward semantic similarity. Results demonstrate that fine-tuned models, particularly Mistral-7B, significantly outperform base models, and soft metrics like BLEU-E achieve much higher correlations with human expert rankings than traditional hard metrics.
Introduction
In fields such as social science and ecology, Fuzzy Cognitive Maps (FCMs) serve as vital tools for modeling how individuals perceive cause-and-effect within complex systems. These maps are signed, weighted directed graphs where nodes represent concepts and edges denote causal influences. For example, a map might illustrate how "Increased rainfall" leads to "Higher crop yield."
The primary challenge lies in the extraction process. Traditionally, building these maps requires intensive manual coding of interviews or documents. While LLMs offer a path toward automation, evaluating their output is difficult. Standard "hard" metrics like the F1 score require exact string matches; if a model identifies "agricultural yield" instead of "crop production," it is penalized despite being semantically correct. This rigidity ignores the "partial correctness" inherent in natural language.
Methodology
To address these challenges, we developed a multi-step pipeline for automated FCM extraction and evaluation:
- Model Selection and Fine-Tuning: We utilized Llama-2, Llama-3, and Mistral-7B. These models were fine-tuned using Low-Rank Adaptation (LoRA) on a specialized dataset of 318 text passages concerning Social-Ecological Systems (SES).
- Soft Evaluation Metrics: We adapted common NLP metrics—BLEU, METEOR, ROUGE, and BLEURT—into "edge-based" versions (e.g., BLEU-E). These measures allow for approximate semantic matches between extracted nodes and edges.
- Partial Positives (PP): Our scoring formula includes "Partial Positives," which reward the model for identifying correct causal links even if the specific direction of the relationship (increase vs. decrease) is incorrect.
- Human-Centric Validation: To establish a "gold standard," we conducted an Elo tournament where human raters compared model-generated extractions. We then calculated the Spearman correlation between automated metrics and these human rankings.
Results and Discussion
Our evaluation revealed that traditional metrics are inadequate for graph extraction tasks. The standard F1 score showed a near-zero correlation (0.016) with human judgment. In contrast, our proposed soft measures, such as BLEU-E, achieved a mean correlation of 0.415, demonstrating their superior alignment with expert perception.
Fine-tuning proved essential for high-quality extraction. As shown in the performance data, the BLEU-E score for the Mistral-7B model nearly doubled following LoRA adaptation, consistently outperforming the Llama variants. Furthermore, the inclusion of Partial Positives was crucial; in many real-world scenarios, identifying that a causal link exists is a significant first step, even if the sign of the link requires further refinement.
Conclusion
This research represents a significant advancement toward "Collective Intelligence"—the ability to aggregate individual mental models into a coherent system-wide map. By moving from rigid "hard" metrics to semantically aware "soft" measures, we can better validate and improve the automated extraction of causal knowledge. Future work will focus on expanding the dataset across different domains and integrating human-in-the-loop feedback to further refine LLM reasoning.
Summary updated