TL;DR
This paper presents three distinct approaches to automatically detect signs of depression in social media text, leveraging BERT models, AutoML, and knowledge graphs. Their combined knowledge graph and textual representation approach achieved a strong performance, ranking 9th out of 31 teams in a shared task, highlighting the potential of integrating structured knowledge with deep learning for mental health analysis.
What Problem Does This Paper Address?
Depression is a serious mental illness with significant consequences if left untreated. Early detection is crucial, and social media has become a common platform for individuals to express their feelings. This paper tackles the challenge of automatically identifying signs of depression within social media text, a task that has become increasingly important with the rise of online communication, especially amplified by events like the COVID-19 pandemic. The goal is to develop accurate predictive models that can assist in early detection, potentially aiding mental health professionals.
Methods
The authors explored three primary methodologies:
- BERT-based Fine-tuning: They fine-tuned several pre-trained BERT variants, including BERT-End-to-End, DistilBERT, and RoBERTa, on the depression detection task. RoBERTa, being trained on more data and longer sequences, was chosen for the final submission due to its superior performance.
- AutoML (autoBOT): They employed autoBOT, an Automated Machine Learning system that iteratively generates and re-weights document representations. This system combines symbolic features (words, characters, POS tags) and sub-symbolic features (contextual and non-contextual sentence embeddings) to create a final representation.
- Knowledge Graph (KG) and Textual Representations: This approach involved generating standalone text and knowledge graph representations. For KG features, they utilized six different embedding models (TransE, RotatE, ComplEx, DistMult, SimpLE, QuatE) trained on WikiData5m. Textual features were derived using Latent Semantic Analysis (LSA) and contextual embeddings from DistilBERT. They then explored two strategies for merging these: concatenation and normalization (CN), and dimensionality reduction via Singular Value Decomposition (SVD) (DR).
Datasets
The primary dataset used in this work was provided by the shared task organizers. It consists of English posts from the Reddit social media platform, labeled into three classes: "not depressed," "moderate," and "severe." The dataset was split into training, development, and testing sets.
Key Insights and Ideas
- Knowledge Graphs Enhance Textual Analysis: The most significant finding is that combining knowledge graph representations with textual features, particularly when dimensionality reduction (DR) was applied, yielded substantially better results than using either approach alone or simple concatenation. This suggests that structured knowledge about concepts can significantly enrich the understanding of textual content for depression detection.
- RoBERTa as a Strong Baseline: While not the top-performing method overall, the fine-tuned RoBERTa model served as a strong baseline, demonstrating the effectiveness of large pre-trained language models for this task.
- AutoML's Iterative Approach: autoBOT showcased an interesting approach to feature engineering, iteratively evolving representations. While it performed well, it was outperformed by the KG approach.
- Importance of Dimensionality Reduction: For the KG approach, applying SVD to reduce the dimensionality of combined KG and textual features led to superior performance compared to simple concatenation, indicating that finding a more compact latent space is beneficial.
Key Results
The paper reports results on both a development set and the official test set.
- Development Set: The Knowledge Graph (KG) approach with dimensionality reduction (DR) achieved the highest Macro F1-score (0.8627) on the development set, significantly outperforming BERT models (best Macro F1 of 0.5287 with RoBERTa) and autoBOT (Macro F1 of 0.5276).
- Official Test Set: The best-performing model submitted by the team was the Third Method (Knowledge Graphs), achieving an Accuracy of 0.6015, Macro Recall of 0.5714, Macro Precision of 0.5149, and a Macro F1-score of 0.5334.
- Competition Ranking: The team ranked 9th out of 31 teams. Their best solution was 4.9% behind the top score in Macro F1 and 1.9% behind in Recall.
Limitations and Future Work
The authors acknowledge that their best solution, while strong, was still a few percentage points behind the leading model. For future work, they propose:
- Ensemble Methods: Developing ensembles of their previously created models to leverage the strengths of different feature types and approaches.
- Larger/More Specific Knowledge Graphs: Exploring larger knowledge graphs like ConceptNet or specialized medical knowledge graphs to potentially improve performance.
- Feature Importance Analysis: Investigating feature importance methods to understand how different feature families contribute to the model's outcomes and gain deeper insights into depression detection.
Summary updated