YaleNLP @ PerAnsSumm 2025: Multi-Perspective Integration via Mixture-of-Agents for Enhanced Healthcare QA Summarization

Dongsuk Jang, Haoxin Li, Arman Cohan

TL;DR

This paper tackles the challenge of summarizing healthcare Q&A forum discussions, which often contain diverse and sometimes conflicting user perspectives. The authors propose a "Mixture-of-Agents" (MoA) framework that combines multiple large language models (LLMs) to identify and summarize these different viewpoints, achieving the second-place rank in the PerAnsSumm 2025 shared task.

What Problem Does This Paper Address?

Healthcare community question-answering (CQA) forums are a popular source for medical advice, but they present a significant challenge for summarization. Unlike curated medical sites, these forums feature a wide array of user-generated content, including personal experiences, anecdotal evidence, and speculative reasoning. This diversity, while potentially enriching, leads to information overload and makes it difficult for users to extract critical insights. Traditional summarization methods often focus on a single "best" answer, failing to capture the multifaceted perspectives present across multiple responses. The PerAnsSumm Shared Task was designed to address this by requiring systems to identify specific perspectives within answers and then generate perspective-specific summaries.

Methods

The researchers explored several approaches:

  • Zero-/Few-Shot Prompting: They utilized large language models (LLMs) like LLaMA-3.3-70B-Instruct and GPT-4o with carefully crafted prompts. For few-shot learning, they compared manually curated examples with automatically selected examples based on sentence-transformer embeddings, finding the latter to be more effective.
  • Mixture-of-Agents (MoA): This core contribution involves an ensemble of LLMs. Different LLMs act as "agents," and their outputs are aggregated. They experimented with 1, 2, and 3-layer MoA architectures, incorporating a "verification" layer in the 2-layer setup and a "hallucination detection" layer in the 3-layer setup. This approach leverages the complementary strengths of diverse models (both open-source and closed-source like GPT-4o) for better medical reasoning and language understanding.
  • QLoRA Supervised Fine-Tuning: They also investigated parameter-efficient fine-tuning of LLaMA-3.3-70B-Instruct using QLoRA.

Datasets

The primary dataset used was the PUMA corpus, provided by the PerAnsSumm shared task. This corpus consists of 3,245 Q&A threads, each with perspective annotations (cause, suggestion, experience, information, question) and reference summaries for each perspective. The data was split into training, validation, and testing sets.

Key Insights and Ideas

  • GPT-4o Dominance: The closed-source GPT-4o model consistently outperformed even the largest open-source LLMs (like LLaMA-3.3-70B-Instruct) in both span identification/classification and perspective-based summarization, even in zero-shot settings.
  • Embedding-Based Few-Shot Learning: Automatically selecting few-shot examples using sentence-transformer embeddings and clustering proved more effective than manually curated examples for LLaMA models, suggesting better generalizability.
  • MoA's Synergy: The Mixture-of-Agents framework significantly improved performance over individual open-source LLMs, demonstrating the benefit of ensembling diverse models. A 2-layer MoA configuration struck a good balance between thoroughness and accuracy.
  • QLoRA Limitations: Surprisingly, QLoRA supervised fine-tuning did not yield performance gains and even degraded results in their experimental setup, possibly due to limited training data or specific configurations.
  • Aggregator Choice Matters: The choice of LLM used as the aggregator in the MoA framework significantly impacts performance.

Key Results

  • Task A (Span Identification/Classification): GPT-4o zero-shot achieved an overall score of 0.5697, outperforming all other single models. Their best MoA configuration (Best 1) achieved 0.5063, a notable improvement over single LLaMA models.
  • Task B (Perspective-Based Summarization): GPT-4o zero-shot achieved an overall score of 0.4180. Their best MoA configuration (Best 1) achieved 0.3688, significantly outperforming LLaMA baselines but still trailing GPT-40.
  • The YaleNLP team's approach ranked second place overall in the PerAnsSumm shared task.

Limitations and Future Work

  • Data Size and Quality: The authors acknowledge that the dataset size and quality might be constraints. Annotations can contain overlaps and ambiguities, complicating accurate labeling.
  • NER Fine-tuning Challenges: Exploratory NER fine-tuning with class weighting did not yield improvements, likely due to the small dataset size and the complexity of labeling overlapping spans in user-generated content.
  • MoA Computational Cost: MoA configurations, especially multi-layer ones, demand significant computational resources.

Future Work: * Investigate larger, more balanced datasets and explore data augmentation techniques. * Develop more robust fine-tuning strategies for imbalanced annotations. * Explore more efficient MoA configurations and aggregator layers. * Further research into prompting construction techniques, including dynamic and retrieval-augmented prompting. * Expand to broader corpora and create synthetic examples for underrepresented perspectives.

Summary updated

Abstract

Automated summarization of healthcare community question-answering forums is challenging due to diverse perspectives presented across multiple user responses to each question. The PerAnsSumm Shared Task was therefore proposed to tackle this challenge by identifying perspectives from different answers and then generating a comprehensive answer to the question. In this study, we address the PerAnsSumm Shared Task using two complementary paradigms: (i) a training-based approach through QLoRA fine-tuning of LLaMA-3.3-70B-Instruct, and (ii) agentic approaches including zero- and few-shot prompting with frontier LLMs (LLaMA-3.3-70B-Instruct and GPT-4o) and a Mixture-of-Agents (MoA) framework that leverages a diverse set of LLMs by combining outputs from multi-layer feedback aggregation. For perspective span identification/classification, GPT-4o zero-shot achieves an overall score of 0.57, substantially outperforming the 0.40 score of the LLaMA baseline. With a 2-layer MoA configuration, we were able to improve LLaMA performance up by 28 percent to 0.51. For perspective-based summarization, GPT-4o zero-shot attains an overall score of 0.42 compared to 0.28 for the best LLaMA zero-shot, and our 2-layer MoA approach boosts LLaMA performance by 32 percent to 0.37. Furthermore, in few-shot setting, our results show that the sentence-transformer embedding-based exemplar selection provides more gain than manually selected exemplars on LLaMA models, although the few-shot prompting is not always helpful for GPT-4o. The YaleNLP team's approach ranked the overall second place in the shared task.
BibTeX
@inproceedings{jang2025yalenlp,
  author = {Dongsuk Jang and Haoxin Li and Arman Cohan},
  title = {{YaleNLP @ PerAnsSumm 2025: Multi-Perspective Integration via Mixture-of-Agents for Enhanced Healthcare QA Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure