Medifact at PerAnsSumm 2025: Leveraging Lightweight Models for Perspective-Specific Summarization of Clinical Q&A Forums

Nadia SaeedNAACL 2025

TL;DR

This paper introduces "Medifact," a novel hybrid framework for summarizing clinical Q&A forum responses by first classifying them into specific perspectives (e.g., cause, suggestion) and then generating multi-perspective summaries. By combining weak supervision with Snorkel, SVMs, and fine-tuned transformer models, Medifact achieves efficient and accurate summarization, securing a top 12 position in the PerAnsSumm 2025 challenge.

What Problem Does This Paper Address?

Healthcare Community Question-Answering (CQA) forums are a rich source of medical information, but the sheer volume and diversity of responses can be overwhelming. Traditional summarization methods often focus on a single "best" answer, failing to capture the nuanced perspectives present across multiple user contributions. This can lead to incomplete or biased information, hindering effective decision-making for users seeking health advice. The PerAnsSumm 2025 challenge specifically targets this gap by requiring perspective-aware summarization of clinical Q&A forums, aiming to provide users with structured, multi-faceted insights.

Methods

Medifact employs a two-stage hybrid approach:

  • Task A: Perspective Classification: This stage aims to categorize each user response into predefined perspectives (e.g., EXPE, INFO, CAUS, SUGG, QUES). It utilizes a three-stage pipeline:
  • Weak Supervision with Snorkel: Leverages rule-based "labeling functions" (LFs) to generate probabilistic labels, reducing the need for extensive manual annotation.
  • Supervised Learning with SVM: Employs Support Vector Machines (SVMs) with sentence embeddings derived from transformer models to refine classification.
  • Zero-Shot Learning (ZSL) with Transformers: Acts as a fallback mechanism for cases where Snorkel and SVM are uncertain, using transformer models (like BART) to classify text based on semantic similarity to predefined categories without prior task-specific training.
  • Task B: Perspective-Aware Summarization: This stage generates structured summaries from the classified responses. It uses a two-step process:
  • Extractive Summarization (BART): Selects key sentences from classified perspectives.
  • Abstractive Refinement (Pegasus): Further refines these extracted sentences to improve fluency and coherence.

The overall framework is designed to be computationally efficient by fine-tuning pre-trained models rather than training from scratch.

Datasets

The paper primarily refers to the PerAnsSumm Shared Task - CL4Health@NAACL 2025 dataset, which is designed for analyzing multi-perspective responses in Community Question Answering (CQA). The dataset includes questions, multi-perspective answers, and is used for both classification and summarization tasks. The authors also mention using a dataset labeled with five perspective categories (EXPE, INFO, CAUS, SUGG, QUES) for training their models.

Key Insights and Ideas

  • Hybrid Approach is Key: Combining weak supervision (Snorkel) with machine learning (SVM) and deep learning (transformers) provides a robust and adaptable solution, especially for domains with limited labeled data.
  • Weak Supervision for Efficiency: Snorkel's ability to generate probabilistic labels from heuristic rules significantly reduces the manual annotation burden, making the system more practical.
  • Zero-Shot Learning as a Smart Fallback: The ZSL component is crucial for handling ambiguous or unseen responses, ensuring that classification is attempted even when other methods abstain.
  • Two-Stage Summarization for Quality: The combination of extractive (BART) and abstractive (Pegasus) summarization effectively balances capturing key information with generating fluent and coherent summaries.
  • Leveraging Pre-trained Models: Fine-tuning pre-trained transformer models (BART, Pegasus) is a resource-efficient strategy that yields high performance without the need for massive computational resources.

Key Results

  • Task A (Classification): The FewShot-SVM+Snorkel+BART model achieved the highest Weighted F1 score of 0.8361. The FewShot-LR+Snorkel+Hybrid (BART+PEGASUS) model also performed competitively with an F1 score of 0.7961 and the best proportional match score (0.7373). Rule-based approaches (RegEx) underperformed significantly.
  • Task B (Summarization): The FewShot-SVM+Snorkel+BART model achieved the highest ROUGE-1 score of 0.3485. The FewShot-LR+Snorkel+Hybrid (BART+PEGASUS) model showed superior factuality (0.2897).
  • Overall Performance: The FewShot-SVM+Snorkel+BART model emerged as the best-performing approach with a combined average score of 0.4077.
  • Challenge Ranking: Medifact secured a position among the top 12 teams in the PerAnsSumm 2025 Shared Task.

Limitations and Future Work

  • Limitations:
    • Weak supervision relies on heuristic rules, which can introduce bias or inconsistencies.
  • While the model achieved good Weighted F1 scores, Macro F1 scores suggest room for improvement in handling class imbalances.
  • Strict matching precision and F1 scores indicate challenges in reducing false positives in span identification.
  • Lower ROUGE and BLEU scores for summarization suggest potential for more accurate and concise text generation.
  • Factual consistency metrics also indicate areas for improvement.
  • Future Work:
    • Explore lightweight architectures, efficient fine-tuning methods (e.g., LoRA, quantization), and retrieval-augmented generation (RAG) to handle unseen perspectives.
  • Focus on enhancing precision in matching.
  • Optimize summarization coherence.
  • Strengthen factual alignment in summaries.
  • Develop more scalable, adaptable, and cost-effective AI solutions, moving beyond reliance on commercial LLMs.

Summary updated

BibTeX
@inproceedings{saeed2025medifact,
  author = {Nadia Saeed},
  title = {{Medifact at PerAnsSumm 2025: Leveraging Lightweight Models for Perspective-Specific Summarization of Clinical Q\&A Forums}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure