TL;DR
This paper introduces KoACD, the first large-scale dataset of cognitive distortions specifically for Korean adolescents, addressing a critical gap in mental health research. They developed a novel multi-LLM negotiation framework with role-switching to improve the accuracy and consistency of classifying these distortions, demonstrating its effectiveness in generating high-quality synthetic data.
What Problem Does This Paper Address?
Cognitive distortions are maladaptive thought patterns linked to mental health issues like depression and anxiety, particularly prevalent in adolescents due to their ongoing cognitive and emotional development. While existing research has focused on adult, English-language datasets, there's a significant lack of resources for adolescent populations, especially in non-English languages. This gap hinders the development of targeted interventions and a deeper understanding of how these distortions manifest in younger individuals. The paper aims to bridge this gap by creating a comprehensive dataset and a robust methodology for identifying and analyzing cognitive distortions in Korean adolescents.
Methods
The core of the paper's methodology lies in its innovative multi-LLM negotiation framework for cognitive distortion classification. This approach involves two LLMs (Gemini 1.5 Flash and GPT-40 mini) acting as an "Analyzer" and an "Evaluator" in a multi-round negotiation. The Analyzer identifies potential cognitive distortions, and the Evaluator provides feedback. Crucially, the models switch roles and engage in iterative feedback over up to five rounds to refine the classification, reducing bias and improving label consistency. This negotiation process is designed to handle the nuances of adolescent language and the complexities of cognitive distortions.
To address data limitations and imbalances, the authors employed two synthetic data generation methods:
- Cognitive Clarification: Rephrasing original text into clearer, more structured forms while preserving meaning and emotional tone.
- Cognitive Balancing: Reconstructing and reorganizing "Unknown" or underrepresented distortion data to balance the dataset.
Finally, an independent evaluation using Anthropic's Claude 3 Haiku was performed to validate the LLM-generated classifications.
Datasets
The primary dataset introduced in this paper is KoACD (Korean Adolescent Dataset for Cognitive Distortion).
- Source: Publicly accessible Q&A posts from NAVER Knowledge iN, specifically from five major adolescent counseling organizations.
- Size: Initially collected 69,925 questions, refined to 37,124 after preprocessing.
- Content: Free-form text written by adolescents, often containing spelling errors and informal language.
- Distortions: Covers 10 predefined cognitive distortions.
The paper also references existing datasets for cognitive distortion detection (Table 1), noting their limitations in scale, language, and target demographic.
Key Insights and Ideas
- Adolescent Language is Nuanced: The study highlights that while LLMs can detect explicit linguistic patterns, human evaluators are superior at understanding the implicit reasoning and context-dependent nuances crucial for accurate cognitive distortion identification in adolescents.
- Multi-LLM Negotiation is Powerful: The role-switching and iterative feedback mechanism in the negotiation framework significantly improves the reliability and consistency of cognitive distortion classification compared to single-pass LLM analysis.
- Synthetic Data is Crucial: Generating synthetic data through clarification and balancing is essential for creating a comprehensive and balanced dataset, especially when dealing with imbalanced categories and the informal nature of adolescent text.
- Context is King: The paper emphasizes that understanding the broader context of a sentence is vital for accurate cognitive distortion analysis, a task where human evaluators still hold an advantage.
Key Results
- KoACD Dataset: The paper presents KoACD, the first large-scale dataset of its kind, containing 108,717 instances after synthetic data generation.
- LLM vs. Human Performance: While LLMs achieved good performance, human evaluators demonstrated higher accuracy, particularly for distortions requiring deeper contextual reasoning. LLMs excelled at detecting explicit patterns like "Should Statements."
- Synthetic Data Evaluation: The synthetic data generated through cognitive clarification and balancing was evaluated by both LLMs and humans. Human evaluation showed higher accuracy for the cognitive balancing method, while LLM evaluation favored cognitive clarification.
- Distribution of Distortions: The synthetic data generation methods helped balance the distribution of the 10 cognitive distortions, with "Emotional Reasoning" and "Discounting the Positive" being the most frequent after generation.
Limitations and Future Work
Limitations: * Blurred Distinctions: The boundaries between some cognitive distortion types are inherently blurred, making classification challenging and leading to potential discrepancies between models and human raters. A multi-label classification method might be beneficial. * "Unknown" Class: The negotiation process sometimes results in an "Unknown" classification due to the inability to fit data within the ten categories, requiring further research into more accurate detection methodologies. * Limited Human Evaluation Data: While the dataset is large, the amount of data reviewed by human raters was relatively small, potentially introducing subjectivity and inconsistency.
Future Work: * Fine-tuning LLMs: Focus on fine-tuning LLMs with adolescent-specific data to enhance contextual understanding. * Improved Algorithms: Develop algorithms that better distinguish cognitive distortions, mitigate biases, and improve both balance and accuracy in detection. * Multi-label Classification: Explore multi-label classification methods and more refined criteria to handle questions with multiple distortions. * Standardized Human Evaluation: Secure more human evaluation data and develop more precise evaluation standards to increase reliability.
Summary updated