Abstract
Large Language Models (LLMs) frequently struggle to capture the global causal "backbone" of complex narratives, often relying on distributional word patterns rather than logical event sequences. This paper introduces a multi-agent framework that utilizes specialized "semantic relation experts"—focusing on temporal, discourse, conditional, and commonsense dimensions—to collaboratively construct explicit causal event graphs. By forcing the model to represent causality through a structured graph before making predictions, the framework significantly improves performance and explainability in downstream reasoning tasks like event forecasting and likelihood estimation.
Introduction
While LLMs excel at identifying local patterns and generating fluent text, they often fail at "global" event reasoning—the ability to understand how a sequence of events logically unfolds over time. For instance, while a model might recognize that earthquakes cause damage, it may fail to identify the specific logical chain where "infrastructure damage" combined with "resource shortages" triggers a "request for international aid."
Most existing models suffer from "transitive incoherence," where they incorrectly chain events based on statistical co-occurrence rather than true cause-effect relationships. To bridge this gap, this research proposes a method to explicitly represent causality through a structured graph, moving beyond simple distributional relations to deep causal connections.
Methodology: Collaborative Causal Graph Generation
The core innovation of this work is a multi-agent framework designed to decompose the complex task of causal reasoning into specialized semantic dimensions. Instead of a single monolithic prompt, the system employs four specialized agents:
- Temporal Expert: Ensures that a proposed cause strictly precedes its effect in time.
- Discourse Expert: Identifies causal links by tracking shared entities (actors or objects) across event descriptions.
- Conditional Expert: Performs counterfactual reasoning, evaluating whether an effect would still occur if the cause were removed.
- Commonsense Expert: Leverages background knowledge to fill in "missing links" that connect two explicit events but are not stated in the text.
The Debate and Consolidation Process
These experts do not work in isolation. They engage in a multi-round debate where they generate initial findings, reflect on the arguments of other experts, and revise their views. This iterative process allows the agents to resolve contradictions—for example, a Temporal Expert might realize a link is valid after the Discourse Expert highlights a shared entity that establishes context.
A final Causality Judge (an LLM arbiter) summarizes the discussion and consolidates the findings into a directed causal graph. To apply these graphs to reasoning tasks, the authors introduce Causal Graph-based Event Likelihood (CGEL), a method that evaluates the likelihood of a new query event by attempting to integrate it logically as a leaf node within the established graph.
Experimental Evaluation
The framework was evaluated across three primary benchmarks:
- CRAB (Causal Reasoning Assessment Benchmark): Used for intrinsic evaluation of causal link identification.
- Annotated NYT Corpus: A curated set of 520 news-query pairs used to evaluate Explainable Event Likelihood (EEL).
- ForecastQA: A benchmark for predicting future events based on historical news data.
Key Results
- Graph Accuracy: Using GPT-4o, the collaborative approach achieved 79.27% Balanced Accuracy in identifying causal links, a significant improvement over the 70.86% achieved by direct prompting.
- Event Forecasting: On the ForecastQA dataset, the graph-based approach reached 62.7% accuracy, outperforming the standard GPT-4 baseline (51.3%) and approaching the performance of specialized, fine-tuned BERT models (67.4%).
- Next Event Prediction: In multiple-choice "cloze" tasks, the framework reached 61.0% accuracy, nearly doubling the performance of one-shot baselines (38.0%).
Discussion and Insights
Several key insights emerged from the study. First, the separation of concerns—breaking causality into specific dimensions—significantly reduces the cognitive load on the LLM. Second, the debate mechanism proved crucial; the authors noted that the Temporal Expert, while initially the weakest, gained the most accuracy and persuasiveness after hearing from other experts.
Furthermore, the use of a graph backbone provides a level of explainability that standard LLM outputs lack. Instead of a simple binary prediction, the model provides a traceable causal chain. These explanations were rated as more coherent and informative by human evaluators compared to standard one-shot generations.
Limitations and Future Work
Despite its success, the framework faces challenges regarding computational cost. The multi-agent debate process costs approximately $0.35 per context, significantly higher than the $0.01 cost of a direct prompt. Future research will focus on pruning unnecessary discussion rounds to improve efficiency.
Additionally, the system remains bounded by the base LLM’s internal logic. If the underlying model possesses a fundamental bias about a real-world event, the experts may reach a consensus on an incorrect conclusion. Finally, the current model treats causality as a binary link; future iterations aim to incorporate causal strength to distinguish between necessary causes and those that merely increase the probability of an effect.