WisPerMed @ PerAnsSumm 2025: Strong Reasoning Through Structured Prompting and Careful Answer Selection Enhances Perspective Extraction and Summarization of Healthcare Forum Threads

Tabea Pakull, Hendrik Damm, Henning Schäfer, Peter Horn, Christoph FriedrichNAACL 2025

TL;DR

This paper tackles the challenge of summarizing healthcare forum discussions by extracting and categorizing different perspectives (cause, suggestion, experience, question, information). They developed a novel approach using structured prompting and careful answer selection with LLMs, achieving the highest overall ranking in the PerAnsSumm 2025 shared task.

What Problem Does This Paper Address?

Healthcare forums are rich sources of patient experiences and medical advice, offering diverse viewpoints. However, traditional summarization methods often focus on a single "best" answer, neglecting the multi-perspective nature of these discussions. This leads to incomplete summaries that fail to capture the full spectrum of information. The authors aim to address this by developing methods for perspective-aware summarization, where different types of information are identified and summarized separately.

Methods

The paper tackles two sub-tasks:

  • Span Identification and Classification: This involves identifying specific text spans within answers that correspond to one of five perspectives (cause, suggestion, experience, question, information). The authors experimented with various models, including encoder-based (DeBERTa-v3-large) and decoder-only LLMs (Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct). They explored fine-tuning, instruction-tuning, and prompt-based (zero-shot and few-shot) paradigms. Notably, they found that models with enhanced reasoning capabilities, like DeepSeek-R1, performed well.
  • Perspective-Based Summarization: For each identified perspective, the goal is to generate a concise summary. The key innovation here is the use of an "answer pre-classification" methodology. This involves training multi-label classifiers to label answer spans with their respective perspectives before feeding them into a summarization model (Mistral-7B-v0.3). This ensures the summarization model is trained only on relevant data for each perspective, leading to improved relevance and factuality.

Datasets

The primary dataset used is derived from the L6 Yahoo! Answers CQA repository, filtered for health-related content. It comprises 3,245 question threads and 10,288 individual answers, split into training, validation, and test sets. The dataset is annotated with spans corresponding to the five perspectives.

Key Insights and Ideas

  • Perspective Matters: Simply summarizing the "best" answer misses crucial information. Acknowledging and separating different perspectives is key to comprehensive healthcare forum summarization.
  • Structured Prompting is Powerful: Using carefully designed system and user prompts, especially in a conversational format, significantly enhances LLM performance in tasks like span extraction. This helps models understand and adhere to complex instructions.
  • Pre-classification Boosts Summarization: Filtering and labeling answers by perspective before summarization dramatically improves the quality of the generated summaries, leading to better relevance and factuality.
  • Reasoning Capabilities are Crucial: Models with stronger reasoning abilities (e.g., DeepSeek-R1) show superior performance in span extraction, even when dealing with complex contextual cues.
  • Data Imbalance is a Challenge: Underrepresented perspectives (like "cause" and "question") in the dataset negatively impact classifier performance and highlight a broader challenge in creating balanced datasets for such tasks.
  • Hallucinations Remain a Problem: LLMs still struggle with generating outputs that are perfectly faithful to the source text, a critical issue in healthcare applications.

Key Results

  • Overall Ranking: The WisPerMed team achieved the highest overall ranking in the PerAnsSumm 2025 shared task.
  • Task A (Span Identification): DeepSeek-R1 in a zero-shot setting achieved a high proportional recall (0.738) and F1-Score (0.676). However, strict boundary alignment remained challenging, with a strict F1-Score of only 0.196.
  • Task B (Summarization): The approach using Mistral-7B-v0.3 with answer pre-classification achieved the best overall performance, with the highest ROUGE-1 (0.452) and balanced factuality (SummaC: 0.296).

Limitations and Future Work

  • Data Imbalance: The underrepresentation of certain perspectives in the dataset negatively impacts classifier performance and robustness. Future work could explore methods to address this imbalance.
  • No Penalty for "Useless" Summaries: The current evaluation doesn't penalize generating summaries for perspectives without supporting evidence. Incorporating such penalties could lead to more accurate utility assessments.
  • Automatic Metrics Limitations: Current automatic metrics may not fully capture clinical relevance and interpretability, potentially leading to an incomplete assessment of model performance.
  • Hallucinations: The tendency of LLMs to hallucinate or introduce extraneous content remains a significant challenge, especially for larger decoder-only models. Further research into improving model fidelity is needed.

Summary updated

BibTeX
@inproceedings{pakull2025wispermed,
  author = {Tabea Pakull and Hendrik Damm and Henning Schäfer and Peter Horn and Christoph Friedrich},
  title = {{WisPerMed @ PerAnsSumm 2025: Strong Reasoning Through Structured Prompting and Careful Answer Selection Enhances Perspective Extraction and Summarization of Healthcare Forum Threads}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure