The Wisdom of Agent Crowds: A Human-AI Interaction Innovation Ignition Framework

Senhao Yang, Qiwen Cheng, Ruiqi Ma, Liangzhe Zhao, Zhenying Wu, Guangqiang YuarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces a human-AI collaborative framework designed to spark innovation in high-risk financial decision-making. By combining multi-agent "brainwriting" with a Belief-Desire-Intention (BDI) model, the researchers created a system that aligns AI collective wisdom with human goals while using novel entropy-based metrics to prove that structured AI collaboration significantly boosts idea diversity.

What Problem Does This Paper Address?

While Large Language Models (LLMs) are increasingly used in multi-agent systems (MAS), they face two major hurdles in high-stakes fields like finance: alignment and evaluation.

First, fully autonomous systems often struggle to stay aligned with complex human intentions, leading to "black box" decisions that experts can't trust. Second, there is a lack of objective ways to measure the quality of "brainstorming" sessions. Most current evaluations rely on subjective "LLM-as-a-judge" methods or human scoring, which are prone to bias. Furthermore, humans interacting with these complex systems often face a high "cognitive load"—they get overwhelmed by the sheer volume of AI-generated text.

Methods

The authors propose a framework called System X, built on several innovative pillars:

  • BDI-Based Interaction: Rooted in the Belief-Desire-Intention (BDI) theory of agency, the system recognizes user needs to plan tasks, ensuring the AI agents work toward the human's actual goals.
  • The "Brainwrite" Workflow: Unlike traditional brainstorming where agents might influence each other too early (the "bandwagon effect"), this method has agents record ideas independently before sharing and iterating.
  • Cothinker Module: This interactive assistant provides two levels of support: "System-1" (intuitive, quick creative directions) and "System-2" (deliberate, in-depth logical analysis).
  • Cognitive Load Reduction: The system uses real-time structured summaries and mind maps to help humans digest information without hitting "mental redline."
  • K-means-Entropy Evaluation: To objectively measure "wisdom," the authors vectorized AI-generated ideas, clustered them using K-means, and calculated Information Entropy. Higher entropy indicates a more diverse and "wiser" set of ideas.
  • LLM-Based Sentiment Analysis: For qualitative feedback, they used next-token prediction probabilities from models like Qwen2.5 and DeepSeek-R1 to quantify the sentiment of user interviews.

Datasets

The researchers evaluated their framework using:

  • 30 Open-Ended Discussion Topics: Spanning three domains: "mathematical sciences," "finance," and "philosophy."
  • LLM Generated Data: Collected from sessions using various models (Zhipu API's Flash, Air, and Plus versions) under different prompting strategies (Zero-shot vs. Chain-of-Thought).
  • Human Factors Data: 118 questionnaires (58 System Usability Scale (SUS) and 60 NASA Task Load Index (NASA-TLX)) and 10 in-depth interview records from human participants.

Key Insights and Ideas

  • Structure Beats Raw Power: Simply asking an LLM for ideas is less effective than the "Brainwrite" method. Providing background context and forcing independent thought before collaboration improved diversity metrics by a staggering 178.4%.
  • CoT is a Diversity Multiplier: Chain-of-Thought (CoT) prompting didn't just make the agents "smarter"—it made their collective output significantly more diverse compared to zero-shot prompting.
  • The "Sweet Spot" for Agents: The research explored how the number of participants (C) affects output. While more agents generally increase diversity, the architectural design (how they are coordinated) is more crucial than just adding more "bots" to the room.
  • Cognitive Offloading Works: The use of mind maps and structured text was cited by users as a primary factor in reducing the mental effort required to manage a multi-agent team.

Key Results

  • Diversity Gains: The combination of Brainwrite and background prompts yielded the highest entropy scores across all tested domains.
  • Usability: System X achieved "moderate usability" scores on the SUS scale. While users found it effective for complex financial tasks, they noted room for improvement in personalization.
  • Sentiment Analysis: Quantitative analysis of interviews showed a predominantly positive sentiment toward the system, though technical stability (latency) and data security were highlighted as recurring concerns for professional use.
  • Model Performance: In the sentiment analysis task, the fine-tuned Qwen2.5-7B-Guba-Senti model outperformed FinBERT, demonstrating the power of domain-specific fine-tuning for financial social media and interview data.

Limitations and Future Work

The authors are transparent about several constraints:

  • Framework Dependency: The current system relies heavily on the Streamlit framework, which may face stability issues in extremely large-scale or high-concurrency scenarios.
  • Model Sensitivity: The evaluation of "diversity" and "sentiment" is still dependent on the underlying LLMs used for the metrics themselves; further validation across more diverse model families is needed.
  • Future Directions: The team plans to integrate multi-modal features (voice, video, and more advanced visualization) and expand the framework into other high-risk domains like healthcare and education. They also aim to refine the multi-agent coordination to make the system more "dynamic" in adapting to shifting user intentions mid-session.

Summary updated

Abstract

With the widespread application of large AI models in various fields, the automation level of multi-agent systems has been continuously improved. However, in high-risk decision-making scenarios such as healthcare and finance, human participation and the alignment of intelligent systems with human intentions remain crucial. This paper focuses on the financial scenario and constructs a multi-agent brainstorming framework based on the BDI theory. A human-computer collaborative multi-agent financial analysis process is built using Streamlit. The system plans tasks according to user intentions, reduces users' cognitive load through real-time updated structured text summaries and the interactive Cothinker module, and reasonably integrates general and reasoning large models to enhance the ability to handle complex problems. By designing a quantitative analysis algorithm for the sentiment tendency of interview content based on LLMs and a method for evaluating the diversity of ideas generated by LLMs in brainstorming based on k-means clustering and information entropy, the system is comprehensively evaluated. The results of human factors testing show that the system performs well in terms of usability and user experience. Although there is still room for improvement, it can effectively support users in completing complex financial tasks. The research shows that the system significantly improves the efficiency of human-computer interaction and the quality of decision-making in financial decision-making scenarios, providing a new direction for the development of related fields.
BibTeX
@misc{yang2025wisdom,
  author = {Senhao Yang and Qiwen Cheng and Ruiqi Ma and Liangzhe Zhao and Zhenying Wu and Guangqiang Yu},
  title = {{The Wisdom of Agent Crowds: A Human-AI Interaction Innovation Ignition Framework}},
  year = {2025},
  eprint = {2505.06947},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.06947},
  howpublished = {arXiv},
}
Figure