TL;DR
This paper investigates how incorporating external knowledge from sentiment and emotion lexicons can improve transformer-based models for estimating depression symptoms from text. They found that while these lexicons can boost performance, their effectiveness depends on the specific dataset and task, with some lexicons showing more promise than others.
What Problem Does This Paper Address?
Estimating mental health conditions, particularly depression, from text is a growing area of research. While transformer models have shown great promise, they often struggle to capture the nuanced emotional and sentiment-laden language associated with depression. Existing approaches have either relied on pre-transformer era models or haven't fully explored how to effectively integrate external linguistic knowledge (like sentiment or emotion lexicons) into modern transformer architectures for this specific task. This paper aims to bridge this gap by evaluating different lexicon incorporation strategies within transformer models for depression symptom estimation.
Methods
The core of this work lies in how they integrate external knowledge. Instead of modifying the transformer architecture itself, they employ an "input marking" strategy. This involves annotating words in the input text that are present in specific lexicons with special markers (e.g., "@word@"). This approach keeps the model's architecture and vocabulary unchanged, making it a flexible way to inject external information.
They experimented with three types of lexicons:
- AFINN: A sentiment lexicon with words rated on a scale from -5 (negative) to +5 (positive).
- NRC: A word-emotion association lexicon linking words to eight basic emotions and two sentiments.
- SDD: A lexicon specifically curated for depression-related words and phrases.
They also explored two transformer-based models: a standard BERT model and a specialized "MentalBERT" model pre-trained on mental health-related data.
Datasets
The study utilizes two primary datasets:
- DAIC-WOZ: This dataset consists of 189 clinical interviews between a virtual therapist and a participant, with each interview transcribed and annotated with self-assessment depression questionnaires (PHQ-8). This dataset is used for estimating depression levels.
- PRIMATE: This dataset comprises Reddit posts from depression-related communities, annotated with binary labels for specific depression symptoms from the PHQ-9. This dataset is used for symptom-based depression prediction.
Key Insights and Ideas
- Lexicons Matter, But Context is Key: The effectiveness of lexicon incorporation is highly dependent on the dataset and the specific task. What works for clinical interviews might not work as well for social media posts.
- Sentiment and Emotion Outperform Domain-Specific (Sometimes): For the DAIC-WOZ dataset, sentiment (AFINN) and emotion (NRC) lexicons generally showed more consistent improvements than the depression-specific SDD lexicon.
- SDD's Weak Signal: The depression-specific SDD lexicon showed limited impact, possibly because it represented a very small fraction of the words in the DAIC-WOZ dataset and was derived from social media data, which might differ in expression from clinical interviews.
- MentalBERT Shows More Robustness: The MentalBERT model, pre-trained on mental health data, generally benefited more from lexicon incorporation, especially when combining all lexicons, suggesting that domain-specific pre-training can enhance the model's ability to leverage external knowledge.
- Input Marking as Regularization: Even randomly marking words showed some improvement, suggesting that the input marking strategy itself can act as a form of regularization, helping the model generalize better.
Key Results
- DAIC-WOZ: Lexicon incorporation, particularly with AFINN and NRC, led to slight overall improvements in depression estimation for the BERT model. The combination of all lexicons yielded the best results for the MentalBERT model, achieving new state-of-the-art results for depression level estimation.
- PRIMATE: The impact of lexicon incorporation was less pronounced and more inconsistent on this dataset. While some lexicons showed minor improvements for specific symptoms, the overall gains were not as significant as on DAIC-WOZ.
- State-of-the-Art: The paper reports new state-of-the-art results for depression level estimation on the DAIC-WOZ dataset when using the MentalBERT model with combined lexicons.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the DAIC-WOZ dataset, while valuable, is relatively small, making it challenging to train and validate hypotheses robustly. They suggest further validation on larger clinical datasets.
- Annotation Consistency: The PRIMATE dataset's annotation by crowd workers, while verified, may contain inconsistencies that affect model performance.
- SDD Lexicon Effectiveness: The limited impact of the SDD lexicon warrants further investigation into why domain-specific lexicons might not always translate to improved performance, especially when the signal is weak or the source data differs.
- Typed Marking: While simple input marking was effective, exploring more sophisticated "typed marking" strategies (e.g., incorporating the actual sentiment/emotion scores) did not yield additional insights in this study, but could be a direction for future work.
Summary updated