LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation

Zhitao He, Pengfei Cao, Yubo Chen, Kang Liu, Ruopeng Li, Mengshu Sun, Jun Zhao

Introduction

While Large Language Models (LLMs) excel at text generation, they often falter when tasked with deep causal reasoning. A common failure mode is the generation of spurious causal associations, where the model relies on superficial patterns rather than rigorous logic. For instance, given a cause involving favorable sailing conditions, an LLM might generically conclude a fleet "made good time," ignoring specific historical or environmental factors that lead to a different effect.

An example of a complex cause-effect pair requiring specific historical and environmental knowledge to explain correctly.

To address this, we introduce LEGO (Multi-agent Collaborative Framework), which treats LLMs as modular components assigned to specialized roles. This collaborative approach mimics human expert reasoning, moving beyond unidirectional "cause-to-effect" jumps to a more holistic, bidirectional analysis.

The LEGO Framework

The LEGO architecture is divided into two primary modules designed to bridge the gap between raw input and logical explanation.

1. Fine-grained World Knowledge Integration (FWKI)

This module focuses on gathering the necessary facts to link a cause to an effect. It utilizes three specialized agents:

  • Cause and Effect Analysts: These agents perform bidirectional reasoning. Instead of just looking forward from the cause, the Effect Analyst works backward from the result to identify necessary preconditions.
  • Knowledge Master: Acting as a targeted knowledge retrieval system, this agent synthesizes fine-grained facts to answer specific queries from the Analysts, effectively serving as a more efficient alternative to traditional external search methods.

2. Iterative Feedback and Refinement (IFR)

Once knowledge is gathered, the explanation is polished through a multi-turn dialogue:

  • Explainer: Synthesizes the Analysts' findings into a coherent narrative.
  • Critic: Evaluates the draft for "Observation Feedback" (ensuring it doesn't just restate the input) and "Commonsense Feedback" (verifying logical consistency).

The agents iterate until the explanation meets the required standards of rigor and conciseness.

Experimental Evaluation

The framework was tested on two major benchmarks: WIKIWHY, which requires multi-hop reasoning grounded in Wikipedia, and e-CARE, which focuses on conceptual causal facts.

Key Results

  • Performance Gains: On the WIKIWHY dataset, LEGO achieved an F1 score of 0.666, significantly outperforming standard GPT-3.5 (0.615) and search-augmented baselines like ReAct (0.631).
  • Human Preference: In blind evaluations, human judges found LEGO’s explanations satisfactory 75.3% of the time, a notable improvement over the 71.3% baseline.
  • Ablation Studies: Removing either the Knowledge Master or the Critic led to a sharp decline in performance, confirming that the multi-agent synergy is essential for overcoming reasoning shortcuts.

Discussion and Insights

A critical finding of this research is that LLMs often treat causal explanation as a "concatenation" task—simply gluing the cause and effect together with a transition word. LEGO breaks this habit by forcing the model to investigate the underlying mechanics. Furthermore, using an LLM as an internal "Knowledge Master" proved more effective than external search, as the model could better synthesize relevant context than a raw search engine.

While LEGO offers superior accuracy, it does introduce higher computational costs and latency due to the multi-agent interactions. Future work will focus on optimizing these interactions to reduce noise and improve efficiency in non-linear causal scenarios.

Summary updated

Abstract

Causality Explanation Generation refers to generate an explanation in natural language given an initial cause-effect pair. It demands rigorous explicit rationales to demonstrate the acquisition of implicit commonsense knowledge, which is unlikely to be easily memorized, making it challenging for large language models since they are often suffering from spurious causal associations when they encounter the content that does not exist in their memory. In this work, we introduce LEGO, a Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for causality explanation generation. Specifically, we treat LLM as character malleable LEGO block and utilize role-playing to assign specific roles to five LLMs. We firstly devise a Fine-grained World Knowledge Integration Module to augment information about tasks for alleviating the phenomenon of spurious causal associations. Then, we leverage an Iterative Feedback and Refinement Module to improve the generated explanation by multi-aspect feedback. Extensive experiments on widely used WIKIWHY and e-CARE datasets show the superiority of our multi-agent framework in terms of reasoning about the causality among cause and effect.
BibTeX
@inproceedings{he2023lego,
  author = {Zhitao He and Pengfei Cao and Yubo Chen and Kang Liu and Ruopeng Li and Mengshu Sun and Jun Zhao},
  title = {{LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.613},
}
Figure