TL;DR
This study explores the use of Large Language Models (LLMs) like GPT-4o and Claude 3.5 Sonnet to automatically classify patient complaints in primary care, aiming to overcome the logistical challenges of manual analysis. The LLMs showed promising accuracy, with Claude 3.5 Sonnet performing particularly well, suggesting AI can significantly enhance healthcare quality improvement by identifying systemic issues more efficiently.
What Problem Does This Paper Address?
Healthcare organizations collect patient complaints as a vital source of feedback for improving quality and patient safety. However, manually categorizing and analyzing these complaints is a time-consuming and logistically challenging task. This bottleneck prevents organizations from fully leveraging the insights hidden within this data, often leading to a reactive approach to problem-solving rather than proactive identification and resolution of systemic issues. This paper addresses this challenge by investigating the potential of advanced AI, specifically LLMs, to automate and improve the accuracy and efficiency of patient complaint classification.
Methods
The researchers employed a cross-sectional study design, analyzing 1816 anonymous patient complaints from seven public primary care clinics in Singapore. These complaints were initially manually coded by trained human coders using the Healthcare Complaint Analysis Tool (HCAT) General Practice (GP) taxonomy. Subsequently, three LLMs – GPT-3.5 turbo, GPT-40 mini, and Claude 3.5 Sonnet – were used to classify the same complaints. The LLMs were used in their off-the-shelf configurations without fine-tuning, employing a zero-shot prompting approach. Prompts were designed to fit specific HCAT fields, providing clear definitions and keywords. The LLMs were tasked with outputting classifications in a JSON format, including a summary of the complaint and its category.
To evaluate performance, standard metrics like accuracy and F1-score were used. Inter-rater agreement was assessed using percentage agreement and Cohen's Kappa (κ) to measure concordance between LLM predictions and human coding. A thematic analysis was also conducted using Claude 3.5 Sonnet to identify recurring complaint themes.
Datasets
The study utilized a dataset of 1816 anonymous patient complaints collected from seven public primary care clinics in Singapore between January 1, 2021, and December 31, 2021. This dataset was used for both manual human coding and subsequent LLM classification. No new benchmark datasets were introduced; instead, the existing HCAT (GP) taxonomy served as the classification framework.
Key Insights and Ideas
- LLMs as a Scalable Solution: The core idea is that LLMs can automate a labor-intensive process, freeing up human resources for higher-level analysis and strategic quality improvement.
- Thematic Insights Beyond Classification: Beyond just categorizing complaints, LLMs can also identify overarching themes, revealing common pain points for patients.
- Nuances in LLM Performance: Different LLMs exhibit varying strengths and weaknesses across different classification tasks, highlighting the need for careful model selection and potential fine-tuning.
- The "Swiss Cheese Model" in Practice: The paper implicitly touches upon the idea that patient complaints often arise from a confluence of systemic issues, much like the "Swiss cheese model" of accident causation, where multiple small failures can lead to a significant event.
Key Results
- Dominant Complaint Categories: The majority of complaints fell under the "management" domain (59.4%), specifically related to "institutional processes" (45.7%).
- Prevalent Themes: The most frequent complaint themes identified by Claude 3.5 Sonnet were long wait times (21.6%), staff attitudes (15.8%), and appointment booking difficulties (10.5%), collectively accounting for nearly half of all complaints.
- LLM Accuracy: LLMs achieved moderate to good accuracy (58.4%-95.5%) in HCAT (GP) field classifications.
- Model Performance: GPT-40 mini generally outperformed GPT-3.5 turbo, except in severity classification. Claude 3.5 Sonnet also showed strong performance, often matching or exceeding GPT-40 mini.
- Concordance: All LLMs demonstrated moderate overall concordance (61.9%-68.8%) with human classifications. High concordance was observed for domain classification.
- Agreement: Claude 3.5 Sonnet and GPT-40 mini showed significantly higher concordance and agreement (Cohen's Kappa) in several HCAT (GP) fields compared to GPT-3.5 turbo.
Limitations and Future Work
- Limitations:
- Zero-Shot Approach: The study used off-the-shelf LLMs without fine-tuning, which may limit their performance.
- Data Imbalance: Class imbalance in certain categories (e.g., major harm) could affect statistical robustness.
- Complaint Ambiguity: Short or ambiguous complaints posed challenges for accurate classification by both humans and LLMs.
- Subjectivity in Severity: Assessing complaint severity and harm can be subjective.
- Truncation: A small percentage of complaints were truncated to fit LLM token limits, potentially losing context.
- Single Labeling: The approach of assigning only one category per field might oversimplify complex complaints with multiple issues.
- Language Bias: Potential biases from "Singapore English" and LLM training data were acknowledged.
- Future Work:
- Fine-tuning and Model Training: Further fine-tuning and model training are recommended to improve LLM accuracy.
- Multilabel Classification: Exploring multilabel classification could provide a more nuanced understanding of complaints with multiple issues.
- Handling Longer Complaints: Developing strategies for handling longer complaints without truncation.
- Exploring Other LLMs: Investigating newer LLMs and different architectures.
- Integration into Workflows: Integrating AI into complaint analysis to facilitate proactive identification of systemic issues and enhance quality improvement.
Summary updated