Abstract
Sungjoon Park, Kiwoong Park, Jaimeen Ahn, Alice Oh. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Suicide is a leading cause of death in the military, particularly in countries with compulsory service like the Republic of Korea. Soldiers often face unique stressors—isolation from support networks, harsh unit climates, and combat exposure—that differ significantly from the general public.
Existing suicide detection research typically focuses on general populations, but these models do not always translate to the military context. Furthermore, collecting data from this demographic is difficult due to privacy concerns and the "closed" nature of military life. The authors sought to bridge this gap by creating a high-quality, ethically sourced dataset of military-related social media posts and testing whether state-of-the-art Natural Language Processing (NLP) could accurately flag soldiers in distress.
Methods and Dataset
The researchers followed a multi-stage pipeline to develop a specialized detection framework. They scraped 2,791 posts from Naver Knowledge iN (a popular Korean Q&A platform) using 58 military-related keywords. To protect privacy, they manually stripped all personally identifiable information (PII), including unit names and user IDs.
Dual-Perspective Annotation
A novel methodological step in this study was the recruitment of two distinct expert groups for annotation:
- External Experts: Psychiatrists and psychotherapists.
- Internal Experts: Military counselors and commanders.
Each post was rated on a scale of 0 (No Risk) to 3 (Imminent Risk). This dual approach allowed the researchers to capture both clinical indicators and domain-specific stressors unique to military life.
Modeling
The team fine-tuned several transformer-based models, including Multilingual BERT, XLM-R, and KoBERT (a BERT model pre-trained specifically on Korean text). They compared these against a standard Convolutional Neural Network (CNN) baseline.
Key Results and Insights
The transformer-based models significantly outperformed the CNN baseline across all classification tasks. KoBERT, which was trained on Korean-specific corpora, consistently provided the best results, highlighting the importance of linguistic nuance in mental health detection.
- Binary Classification (Flagged vs. Not Flagged): KoBERT achieved an F1 score of 0.88 and an accuracy of 91%.
- Urgent Risk Detection: For identifying the most critical cases, the models reached an F1 score of 0.80.
- The "Perspective Gap": A fascinating finding was the disagreement between internal and external experts. Military commanders (internal) were often more sensitive to "low-risk" maladaptation to military life, whereas clinicians (external) were better at identifying "imminent" clinical risks. Models trained on "Internal Expert" labels generally yielded higher F1 scores, suggesting their criteria might be more closely tied to the linguistic markers found in the posts.
Discussion and Future Work
While the study demonstrates the effectiveness of NLP in this domain, it is limited by the dataset size and its specific focus on the South Korean military context. The researchers suggest that future work should incorporate military-specific slang and integrate multi-language data to improve robustness. Crucially, the authors emphasize that such tools should be used as support systems for human intervention programs, rather than standalone diagnostic tools.
Summary updated