No perspective, no perception!! Perspective-aware Healthcare Answer Summarization

Gauri Naik, Sharad Chandakacherla, Shweta Yadav, Md. Shad Akhtar

Abstract

Community Question Answering (CQA) forums serve as a vital resource for individuals seeking healthcare advice, yet the resulting threads are often dense and multifaceted, blending clinical facts with personal anecdotes. Standard summarization techniques often fail to distinguish between these nuances, producing generic outputs that may not meet specific user needs. This paper introduces PUMA (Perspective sUMmarization dAtaset), a novel benchmark for healthcare answer summarization, and PLASMA (Perspective-aware heaLthcare Answer SuMmarizAtion), a controllable model designed to generate summaries tailored to specific perspectives. By utilizing prefix tuning and a unique energy-based loss function, PLASMA ensures high faithfulness and consistency, outperforming much larger models like GPT-4 in human evaluations.

Introduction

When users turn to platforms like Reddit’s r/AskDocs or Yahoo! Answers for health information, they are frequently met with a "wall of text." These responses are a complex mixture of clinical data, personal experiences, suggestions, and follow-up questions. For a user specifically looking for others' experiences or potential causes, a standard, one-size-fits-all summary is often inadequate.

The authors argue that for healthcare summarization to be truly effective, it must be perspective-aware. This paper addresses the challenge of extracting specific types of information from noisy CQA data, ensuring that the summary aligns with the user's particular intent.

The PUMA Dataset

To facilitate research in this area, the authors developed the Perspective sUMmarization dAtaset (PUMA). This benchmark is significantly more specialized than previous healthcare summarization datasets:

  • Source: 3,167 CQA threads comprising approximately 10,000 answers filtered from Yahoo! Answers.
  • Annotation: Each thread was manually annotated with up to five distinct perspective-specific summaries: Cause, Suggestion, Experience, Question, and Information.
  • Scale: The dataset contains 6,193 perspective-aware summaries, providing a robust foundation for training controllable models.

Notably, the authors intentionally excluded a "Treatment" perspective. This was an ethical decision to prevent the model from inadvertently generating or endorsing specific medical prescriptions, which could pose risks if interpreted as professional medical advice.

The PLASMA Model

The proposed architecture, PLASMA, is designed for efficiency and control. It is built upon three primary pillars:

  1. Frozen Foundation: The model utilizes Flan-T5 as its base. By keeping the base weights frozen, the authors ensure the model remains efficient to train while leveraging the power of a large language model.
  2. Prefix Tuning: Rather than full fine-tuning, PLASMA uses "soft prompts." These are a small set of trainable parameters (prefixes) added to the input that guide the model toward the desired perspective.
  3. Energy-Controlled Loss: To ensure the model strictly adheres to the requested perspective, the authors introduced a novel energy-based objective function. This function calculates three specific energy values for every generated summary:
  4. Perspective-specific ($E_p$): Ensures the content matches the intended category (e.g., "Experience").
  5. Anchor-specific ($E_a$): Ensures the summary begins with an appropriate framing phrase (e.g., "In user's experience...").
  6. Tone-specific ($E_t$): Ensures the semantic "vibe" matches the perspective (e.g., an advisory tone for suggestions).

By minimizing these energy values during training, the model is forced to satisfy multiple constraints simultaneously, leading to more reliable and controllable outputs.

Experimental Results

PLASMA was evaluated against five major baselines, including BART, PEGASUS, and standard T5. The results demonstrated significant improvements across all standard metrics (ROUGE, METEOR, BERTScore, and BLEU), with PLASMA showing a performance boost of 1.5% to 21.8% over the next best baseline.

Comparison with GPT-4

In a head-to-head comparison on 50 samples, GPT-4 achieved higher ROUGE scores, which measure n-gram overlap. However, human evaluation revealed a different priority. Human judges rated PLASMA higher than GPT-4 in Fluency, Consistency, Extractiveness, and Faithfulness. This suggests that while GPT-4 may produce more varied prose, PLASMA is more reliable at staying true to the source material and the requested perspective.

Ablation Study

Ablation tests confirmed the necessity of the energy-controlled loss. Removing any single component (perspective, anchor, or tone constraints) resulted in a measurable drop in performance, proving that all three pillars are essential for high-quality, perspective-driven text generation.

Conclusion and Future Work

PLASMA demonstrates that in specialized domains like healthcare, the intent of a summary is as critical as its content. While the model shows great promise, the authors acknowledge limitations, such as data imbalance in the "Question" perspective and the inherent subjectivity of CQA data. Future research will explore the integration of medical knowledge graphs to further enhance the clinical faithfulness of summaries, moving beyond the limitations of forum-based advice.

Summary updated

Abstract

Healthcare Community Question Answering (CQA) forums offer an accessible platform for individuals seeking information on various healthcare-related topics. People find such platforms suitable for self-disclosure, seeking medical opinions, finding simplified explanations for their medical conditions, and answering others' questions. However, answers on these forums are typically diverse and prone to off-topic discussions. It can be challenging for readers to sift through numerous answers and extract meaningful insights, making answer summarization a crucial task for CQA forums. While several efforts have been made to summarize the community answers, most of them are limited to the open domain and overlook the different perspectives offered by these answers. To address this problem, this paper proposes a novel task of perspective-specific answer summarization. We identify various perspectives, within healthcare-related responses and frame a perspective-driven abstractive summary covering all responses. To achieve this, we annotate 3167 CQA threads with 6193 perspective-aware summaries in our PUMA dataset. Further, we propose PLASMA, a prompt-driven controllable summarization model. To encapsulate the perspective-specific conditions, we design an energy-controlled loss function for the optimization. We also leverage the prefix tuner to learn the intricacies of the health-care perspective summarization. Our evaluation against five baselines suggests the superior performance of PLASMA by a margin of 1.5-21% improvement. We supplement our experiments with ablation and qualitative analysis.
BibTeX
@inproceedings{naik2024no,
  author = {Gauri Naik and Sharad Chandakacherla and Shweta Yadav and Md. Shad Akhtar},
  title = {{No perspective, no perception!! Perspective-aware Healthcare Answer Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.942},
}
Figure