TL;DR
This paper introduces a novel two-stage "Classifier-Refiner Architecture" (CRA) to improve the classification of user responses in community health forums. By first classifying snippets and then refining uncertain classifications with retrieval-augmented examples, their approach significantly boosts accuracy, outperforming single-pass methods and achieving top-tier results in the PerAnsSumm challenge.
What Problem Does This Paper Address?
Community Question-Answering (CQA) platforms are invaluable for sharing health-related information, but the user-generated nature of these discussions, coupled with the inherent complexity and subjectivity of natural language, makes automatic classification a significant challenge. Specifically, the PerAnsSumm task aims to extract and classify "perspective spans" within user answers. Traditional single-pass classification systems often struggle with nuanced language, leading to misclassifications or missed information, which can have serious consequences in the healthcare domain. This paper tackles the problem of accurately categorizing diverse user perspectives within health-related CQA discussions.
Methods
The core innovation is the Classifier-Refiner Architecture (CRA), a two-stage framework:
- Classifier Stage: A language model (like GPT-4o) is prompted to segment user responses into meaningful snippets and assign an initial perspective label along with a confidence score. If the model is uncertain, it flags the snippet as "NOT_CONFIDENT" and provides a reason.
- Refiner Stage: This stage is triggered only for snippets where the Classifier lacked confidence. It employs a retrieval-augmented generation (RAG) mechanism. Specifically, it retrieves the two most similar examples from the training set using a sentence similarity model (all-MiniLM-L6-v2). These retrieved examples are then used as few-shot prompts to help the Refiner confirm or reassign the category.
The methodology also incorporates:
- Instruction-driven classification: Clear instructions guide the models.
- Tone definitions: Explicitly defining the "tone" associated with each category (e.g., "Personal, Narrative" for EXPERIENCE) helps disambiguate similar perspectives.
- Chain-of-Thought (CoT) prompting: Encourages intermediate reasoning steps for more robust decision-making.
Datasets
The primary dataset used is PUMA, created for the PerAnsSumm shared task. PUMA was derived from the L6 - Yahoo! Answers Comprehensive Questions and Answers corpus, filtered for healthcare-related content, and further refined and annotated for the PerAnsSumm task. It was split into training (2,236 pairs), validation (959 pairs), and test (50 pairs) sets. The annotations were performed by multiple annotators to ensure reliability and consistency, categorizing spans into five labels: Cause, Suggestion, Experience, Question, and Information.
Key Insights and Ideas
- Iterative Refinement is Key: The two-stage CRA significantly outperforms single-pass methods, demonstrating that a secondary refinement step can effectively handle ambiguity and improve classification accuracy.
- Tone Matters: Explicitly defining the "tone" for each category (e.g., distinguishing between "Informative" and "Personal/Narrative") is crucial for models to differentiate subtle semantic differences.
- CoT Boosts Precision: Chain-of-Thought reasoning, while adding a slight computational cost, leads to a measurable increase in precision, especially for complex or overlapping perspectives.
- LLM Agnosticism (to an extent): The CRA framework shows promise across different LLMs (GPT-4o, Claude 3, o1-preview), highlighting the generalizability of the multi-stage approach.
- Internalized Reasoning vs. Explicit Prompts: Models like o1-preview, which inherently manage CoT reasoning, can achieve high performance without explicit user-guided CoT prompts, simplifying prompt engineering.
Key Results
- The proposed CRA framework achieved an overall score of 0.6090 on the PerAnsSumm shared task, ranking among the top-performing teams.
- The o1-preview model, using the CRA approach with tone definitions, achieved the best overall score of 0.6090, with particularly strong Proportional Recall (0.8406) and Proportional F1 (0.7382) scores.
- The CRA approach consistently improved metrics (macro-F1, weighted-F1, strict, and proportional) compared to single-pass methods across various LLMs.
- The Refiner stage was triggered for 21 out of 1039 snippets, indicating it efficiently targeted only the most ambiguous cases.
Limitations and Future Work
- Data Dependency: The CRA's effectiveness heavily relies on the quality and coverage of the training data. If the training set lacks relevant examples for ambiguous snippets, the Refiner might struggle.
- No Factual Validation: The system is designed for classification, not for validating medical claims. Erroneous but plausible suggestions could still be classified.
- Domain Specificity: The current work is limited to a single domain (healthcare) and language. Generalizing to other domains or languages requires further adaptation.
- Inference Time: While efficient by only triggering the Refiner when needed, the iterative nature still adds inference time for borderline cases, which might be a concern for real-time applications.
Future work could involve:
- Enhancing model interpretability.
- Domain-specific fine-tuning for nuanced medical conditions.
- Cross-lingual adaptations.
- Integrating external medical knowledge bases or discourse-level context for deeper inference.
- Extending the framework to broader, multi-turn QA and summarization tasks.
Summary updated