Introduction
In the realm of healthcare, Community Question Answering (CQA) forums serve as vital resources for individuals seeking medical advice and shared experiences. However, these platforms often present a challenge: a single query can elicit a deluge of responses ranging from clinical facts to anecdotal personal stories. Traditional summarization techniques frequently default to the "best-voted" answer, potentially overlooking the diverse and valuable perspectives contained within the broader thread. The YaleNLP team addressed this challenge in the PerAnsSumm 2025 shared task by developing a system capable of identifying and integrating these multi-faceted viewpoints into coherent summaries.
The Challenge of Multi-Perspective Summarization
The PerAnsSumm Shared Task (NAACL 2025) requires systems to navigate the messy, unstructured nature of user-generated healthcare content. The goal is to identify five specific "perspectives"—Information, Cause, Suggestion, Experience, and Question—across multiple answers and generate a summary for each. This task utilizes the PUMA (Perspective sUMmarization dAtaset), which contains 3,245 Q&A threads with detailed perspective annotations.
Methodology: Mixture-of-Agents (MoA)
The core of the YaleNLP approach is a sophisticated Mixture-of-Agents (MoA) framework. Rather than relying on a single large language model (LLM), the team implemented a multi-layer pipeline designed to iteratively refine the output:
- Layer 1 (Proposers): Multiple open-source models, including LLaMA-3.3-70B and Qwen-2.5-72B, generate initial candidate summaries.
- Layer 2 (Verification): An intermediate model reviews and refines these candidates for consistency and clarity.
- Layer 3 (Hallucination Detection): A final verification step filters out unsupported medical claims to ensure safety.
- Aggregator: A high-capacity model (e.g., LLaMA-3.3-70B-Instruct) fuses the refined outputs into a final, coherent response.
Beyond MoA, the team explored Embedding-Based Selection for few-shot prompting. By using sentence-transformers to cluster training data, they could automatically select the most structurally relevant examples for a given query, outperforming manual selection. Interestingly, the team found that QLoRA fine-tuning actually degraded performance compared to zero-shot prompting, likely due to the noise and limited size of the training set.
Experimental Results
The MoA approach demonstrated significant performance gains, particularly for open-source models. In Task A (Span Identification), a 2-layer MoA setup boosted the LLaMA baseline from a score of 0.40 to 0.51. In Task B (Summarization), the MoA framework improved LLaMA’s performance by 32%, reaching a score of 0.37 and narrowing the gap with closed-source models like GPT-4o.
As shown in the performance charts, the "Multi Proposer" configuration consistently outperformed the "Single Proposer" baseline across all metrics. The results also highlight that performance typically peaks at two layers, suggesting a point of diminishing returns for additional computational complexity in this specific task.
Discussion and Conclusion
The YaleNLP team’s strategy secured second place in the PerAnsSumm 2025 shared task. Their findings emphasize that the choice of an "Aggregator" model is critical; larger models like LLaMA-3.3-70B proved more capable of reasoning over multiple inputs than smaller alternatives.
Future work will focus on addressing data scarcity through synthetic data generation and investigating more efficient aggregator layers to reduce the computational overhead of multi-model ensembles. The success of the MoA framework suggests that integrating diverse model perspectives is a powerful path forward for complex, multi-faceted summarization tasks in the healthcare domain.
Summary updated