UMB@PerAnsSumm 2025: Enhancing Perspective-Aware Summarization with Prompt Optimization and Supervised Fine-Tuning

Kristin Qi, Youxiang Zhu, Xiaohui LiangNAACL 2025

TL;DR

This paper tackles the challenge of summarizing medical advice from community forums, where different perspectives are crucial for understanding. The authors developed a novel approach combining ensemble learning for identifying relevant text spans and a sophisticated Chain-of-Thought prompting strategy, further enhanced by automated prompt optimization and supervised fine-tuning, to generate more accurate and factually consistent summaries.

What Problem Does This Paper Address?

Community question-answering (CQA) platforms, especially in the medical domain, are rich sources of information. However, the sheer volume and diverse viewpoints within answers make it difficult for users to extract accurate and relevant information efficiently. Traditional summarization methods often fail to capture these distinct perspectives, leading to a loss of crucial context. This paper addresses the need for "perspective-aware summarization," which aims to organize and present information by considering the different viewpoints expressed in the CQA threads, making medical information more accessible and understandable.

Methods

The proposed approach has two main components:

  • Perspective Span Identification: To accurately pinpoint the parts of an answer that represent specific perspectives (e.g., Information, Cause, Suggestion, Experience, Question), the authors employed an ensemble learning strategy. This involves integrating the predictions of three pre-trained transformer models (BERT, RoBERTa, and DeBERTa) by averaging their outputs. This ensemble approach leverages the individual strengths of each model to improve overall accuracy.
  • Perspective-Aware Summarization: For generating summaries, the researchers developed a multi-stage process:
  • Chain-of-Thought (CoT) Prompting: This involves breaking down the summarization task into smaller, manageable steps. The CoT strategy incorporates:
  • Keyphrase Extraction: Identifying and extracting important keywords from the identified perspective spans.
  • Keyphrase Integration: Ensuring these extracted keyphrases are included in the generated summary.
  • Guide Information Integration: Using specific instructions related to the perspective's tone, common anchor texts (starting phrases), and a concise definition of the perspective.
  • Prompt Optimization with DSPy: To further refine the CoT prompts, they utilized the DSPy framework. DSPy automates prompt refinement by generating and selecting optimal prompt variants through Bayesian optimization, aiming to improve performance on a composite metric that balances relevance and factuality.
  • Supervised Fine-Tuning (SFT): Finally, they fine-tuned the Llama-3 LLM on domain-specific data using the Low-Rank Adaptation (LoRA) technique. This step helps the model adapt to the nuances of medical CQA content.

Datasets

The study used the CQA threads from medical forums provided for the PerAnsSumm 2025 shared task. The dataset was divided into three parts: training, validation, and test sets. The training set contained 2236 labeled CQA threads with annotated perspective spans and reference summaries, while the validation set had 959 samples for hyperparameter tuning. The test set was kept hidden for final evaluation.

Key Insights and Ideas

  • Ensemble learning boosts span identification: Combining multiple transformer models through averaging significantly improved the accuracy of identifying perspective spans, outperforming individual models in certain aspects.
  • Structured reasoning is key for summarization: The Chain-of-Thought prompting, especially when incorporating keyphrases and specific guidance, demonstrably improved the quality and alignment of generated summaries with references.
  • Automated prompt optimization is powerful: DSPy's ability to iteratively refine prompts led to substantial improvements in both relevance and factuality metrics, showing that automated methods can effectively build upon manual prompt engineering.
  • Fine-tuning adapts LLMs to domain specifics: Supervised fine-tuning of Llama-3 on medical CQA data further enhanced summarization performance, particularly in relevance, suggesting that domain adaptation is crucial for complex tasks.
  • Relevance vs. Factuality trade-off: While SFT significantly improved relevance metrics, its impact on factuality was less pronounced, indicating that fine-tuning primarily enhances content generation aligned with references rather than strictly improving factual accuracy.

Key Results

  • Span Identification: The ensemble model achieved an F1-score of 82.9% on the test set and 83.9% on the validation set for perspective span identification.
  • Summarization:
    • The CoT approach with keyphrases and guide information improved ROUGE-1 by +8.1% and BERTScore by +1.5% over vanilla prompting.
  • DSPy optimization on the CoT prompt led to average improvements of +25.6% on validation and +9.1% on test sets across relevance metrics. Factuality metrics also saw significant gains (+17.0% for AlignScore, +4.7% for SummaC).
  • Combining DSPy optimization with SFT further boosted performance, with ROUGE-L improving by +21.4% and Meteor by +7.0% on the validation set. On the test set, Meteor increased by +3.4% and BLEU by +5.5%.

Limitations and Future Work

The authors acknowledge several limitations:

  • LLM choice: They used Llama-3 without benchmarking against API-based models like GPT-4 or Claude-3, noting that Llama-3 underperformed relative to these advanced models.
  • Metric formulation: The DSPy optimizer's reliance on a balanced metric derived from empirical assumptions might oversimplify the relationships between different evaluation metrics, potentially affecting accuracy.
  • Generalizability of prompt optimization: The generalizability of their prompt optimization strategy remains an open question.
  • Domain specificity: The prompt design is tailored to medical CQA and may not account for variability in other summary types.

Future work could involve comparing their approach with other LLMs, exploring alternative optimizers (e.g., MIPRO with bootstrapped demonstrations or OPRO), and investigating metric-based optimization strategies to further improve alignment with references.

Summary updated

BibTeX
@inproceedings{qi2025umb,
  author = {Kristin Qi and Youxiang Zhu and Xiaohui Liang},
  title = {{UMB@PerAnsSumm 2025: Enhancing Perspective-Aware Summarization with Prompt Optimization and Supervised Fine-Tuning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure