Abstract
Community Question Answering (CQA) forums serve as a vital resource for individuals seeking healthcare advice, yet the resulting threads are often dense and multifaceted, blending clinical facts with personal anecdotes. Standard summarization techniques often fail to distinguish between these nuances, producing generic outputs that may not meet specific user needs. This paper introduces PUMA (Perspective sUMmarization dAtaset), a novel benchmark for healthcare answer summarization, and PLASMA (Perspective-aware heaLthcare Answer SuMmarizAtion), a controllable model designed to generate summaries tailored to specific perspectives. By utilizing prefix tuning and a unique energy-based loss function, PLASMA ensures high faithfulness and consistency, outperforming much larger models like GPT-4 in human evaluations.
Introduction
When users turn to platforms like Reddit’s r/AskDocs or Yahoo! Answers for health information, they are frequently met with a "wall of text." These responses are a complex mixture of clinical data, personal experiences, suggestions, and follow-up questions. For a user specifically looking for others' experiences or potential causes, a standard, one-size-fits-all summary is often inadequate.
The authors argue that for healthcare summarization to be truly effective, it must be perspective-aware. This paper addresses the challenge of extracting specific types of information from noisy CQA data, ensuring that the summary aligns with the user's particular intent.
The PUMA Dataset
To facilitate research in this area, the authors developed the Perspective sUMmarization dAtaset (PUMA). This benchmark is significantly more specialized than previous healthcare summarization datasets:
- Source: 3,167 CQA threads comprising approximately 10,000 answers filtered from Yahoo! Answers.
- Annotation: Each thread was manually annotated with up to five distinct perspective-specific summaries: Cause, Suggestion, Experience, Question, and Information.
- Scale: The dataset contains 6,193 perspective-aware summaries, providing a robust foundation for training controllable models.
Notably, the authors intentionally excluded a "Treatment" perspective. This was an ethical decision to prevent the model from inadvertently generating or endorsing specific medical prescriptions, which could pose risks if interpreted as professional medical advice.
The PLASMA Model
The proposed architecture, PLASMA, is designed for efficiency and control. It is built upon three primary pillars:
- Frozen Foundation: The model utilizes Flan-T5 as its base. By keeping the base weights frozen, the authors ensure the model remains efficient to train while leveraging the power of a large language model.
- Prefix Tuning: Rather than full fine-tuning, PLASMA uses "soft prompts." These are a small set of trainable parameters (prefixes) added to the input that guide the model toward the desired perspective.
- Energy-Controlled Loss: To ensure the model strictly adheres to the requested perspective, the authors introduced a novel energy-based objective function. This function calculates three specific energy values for every generated summary:
- Perspective-specific ($E_p$): Ensures the content matches the intended category (e.g., "Experience").
- Anchor-specific ($E_a$): Ensures the summary begins with an appropriate framing phrase (e.g., "In user's experience...").
- Tone-specific ($E_t$): Ensures the semantic "vibe" matches the perspective (e.g., an advisory tone for suggestions).
By minimizing these energy values during training, the model is forced to satisfy multiple constraints simultaneously, leading to more reliable and controllable outputs.
Experimental Results
PLASMA was evaluated against five major baselines, including BART, PEGASUS, and standard T5. The results demonstrated significant improvements across all standard metrics (ROUGE, METEOR, BERTScore, and BLEU), with PLASMA showing a performance boost of 1.5% to 21.8% over the next best baseline.
Comparison with GPT-4
In a head-to-head comparison on 50 samples, GPT-4 achieved higher ROUGE scores, which measure n-gram overlap. However, human evaluation revealed a different priority. Human judges rated PLASMA higher than GPT-4 in Fluency, Consistency, Extractiveness, and Faithfulness. This suggests that while GPT-4 may produce more varied prose, PLASMA is more reliable at staying true to the source material and the requested perspective.
Ablation Study
Ablation tests confirmed the necessity of the energy-controlled loss. Removing any single component (perspective, anchor, or tone constraints) resulted in a measurable drop in performance, proving that all three pillars are essential for high-quality, perspective-driven text generation.
Conclusion and Future Work
PLASMA demonstrates that in specialized domains like healthcare, the intent of a summary is as critical as its content. While the model shows great promise, the authors acknowledge limitations, such as data imbalance in the "Question" perspective and the inherent subjectivity of CQA data. Future research will explore the integration of medical knowledge graphs to further enhance the clinical faithfulness of summaries, moving beyond the limitations of forum-based advice.
Summary updated