TL;DR
This paper tackles the challenge of summarizing healthcare Q&A forum discussions, which often contain diverse and sometimes conflicting user perspectives. The authors propose a "Mixture-of-Agents" (MoA) framework that combines multiple large language models (LLMs) to identify and summarize these different viewpoints, achieving the second-place rank in the PerAnsSumm 2025 shared task.
What Problem Does This Paper Address?
Healthcare community question-answering (CQA) forums are a popular source for medical advice, but they present a significant challenge for summarization. Unlike curated medical sites, these forums feature a wide array of user-generated content, including personal experiences, anecdotal evidence, and speculative reasoning. This diversity, while potentially enriching, leads to information overload and makes it difficult for users to extract critical insights. Traditional summarization methods often focus on a single "best" answer, failing to capture the multifaceted perspectives present across multiple responses. The PerAnsSumm Shared Task was designed to address this by requiring systems to identify specific perspectives within answers and then generate perspective-specific summaries.
Methods
The researchers explored several approaches:
- Zero-/Few-Shot Prompting: They utilized large language models (LLMs) like LLaMA-3.3-70B-Instruct and GPT-4o with carefully crafted prompts. For few-shot learning, they compared manually curated examples with automatically selected examples based on sentence-transformer embeddings, finding the latter to be more effective.
- Mixture-of-Agents (MoA): This core contribution involves an ensemble of LLMs. Different LLMs act as "agents," and their outputs are aggregated. They experimented with 1, 2, and 3-layer MoA architectures, incorporating a "verification" layer in the 2-layer setup and a "hallucination detection" layer in the 3-layer setup. This approach leverages the complementary strengths of diverse models (both open-source and closed-source like GPT-4o) for better medical reasoning and language understanding.
- QLoRA Supervised Fine-Tuning: They also investigated parameter-efficient fine-tuning of LLaMA-3.3-70B-Instruct using QLoRA.
Datasets
The primary dataset used was the PUMA corpus, provided by the PerAnsSumm shared task. This corpus consists of 3,245 Q&A threads, each with perspective annotations (cause, suggestion, experience, information, question) and reference summaries for each perspective. The data was split into training, validation, and testing sets.
Key Insights and Ideas
- GPT-4o Dominance: The closed-source GPT-4o model consistently outperformed even the largest open-source LLMs (like LLaMA-3.3-70B-Instruct) in both span identification/classification and perspective-based summarization, even in zero-shot settings.
- Embedding-Based Few-Shot Learning: Automatically selecting few-shot examples using sentence-transformer embeddings and clustering proved more effective than manually curated examples for LLaMA models, suggesting better generalizability.
- MoA's Synergy: The Mixture-of-Agents framework significantly improved performance over individual open-source LLMs, demonstrating the benefit of ensembling diverse models. A 2-layer MoA configuration struck a good balance between thoroughness and accuracy.
- QLoRA Limitations: Surprisingly, QLoRA supervised fine-tuning did not yield performance gains and even degraded results in their experimental setup, possibly due to limited training data or specific configurations.
- Aggregator Choice Matters: The choice of LLM used as the aggregator in the MoA framework significantly impacts performance.
Key Results
- Task A (Span Identification/Classification): GPT-4o zero-shot achieved an overall score of 0.5697, outperforming all other single models. Their best MoA configuration (Best 1) achieved 0.5063, a notable improvement over single LLaMA models.
- Task B (Perspective-Based Summarization): GPT-4o zero-shot achieved an overall score of 0.4180. Their best MoA configuration (Best 1) achieved 0.3688, significantly outperforming LLaMA baselines but still trailing GPT-40.
- The YaleNLP team's approach ranked second place overall in the PerAnsSumm shared task.
Limitations and Future Work
- Data Size and Quality: The authors acknowledge that the dataset size and quality might be constraints. Annotations can contain overlaps and ambiguities, complicating accurate labeling.
- NER Fine-tuning Challenges: Exploratory NER fine-tuning with class weighting did not yield improvements, likely due to the small dataset size and the complexity of labeling overlapping spans in user-generated content.
- MoA Computational Cost: MoA configurations, especially multi-layer ones, demand significant computational resources.
Future Work: * Investigate larger, more balanced datasets and explore data augmentation techniques. * Develop more robust fine-tuning strategies for imbalanced annotations. * Explore more efficient MoA configurations and aggregator layers. * Further research into prompting construction techniques, including dynamic and retrieval-augmented prompting. * Expand to broader corpora and create synthetic examples for underrepresented perspectives.
Summary updated