Detecting and Explaining Crisis

Rohan Kshirsagar, Robert R. Morris, Sam Bowman

Abstract

Automated crisis detection in social media is a high-stakes task where a simple binary label is often insufficient for effective triage. This paper presents a modular framework that identifies mental health crises (such as suicide, self-harm, and abuse) and generates human-readable explanations by extracting the specific phrases that triggered the model. By combining neural networks with rationale extraction techniques like LIME and dependency parsing, the system provides actionable insights for human responders.

Introduction

The rise of social media has created new opportunities for suicide prevention and mental health outreach. However, automated classifiers often operate as "black boxes," flagging posts without explaining why. For a human responder, the distinction between suicidal ideation, physical abuse, or an eating disorder is critical for determining the appropriate intervention. This research addresses the challenge of "rationale extraction"—identifying a coherent, concise subset of text that justifies a model's decision.

Methodology

The proposed framework consists of two primary stages: Detection and Explanation.

Crisis Detection

The core detection model is a Bidirectional Gated Recurrent Unit (Bi-GRU) RNN, chosen for its ability to capture contextual dependencies in text sequences. An integrated attention mechanism allows the model to weigh specific words more heavily during prediction.

Rationale Extraction

To generate explanations, the authors tested three "seeding" methods to identify influential words:

  1. Logistic Coefficients: Using weights from a baseline logistic regression.
  2. Neural Attention: Leveraging weights from the RNN’s internal attention layer.
  3. LIME (Local Interpretable Model-agnostic Explanations): A technique that perturbs input to observe effects on specific predictions.

The final explanation is not just a collection of words but a grammatically coherent phrase. The system identifies the most important sentence, applies a dependency parser, and extracts the subtree associated with the highest-weighted seed word.

Comparison of explanation methods against human-annotated 'gold' standards.

Data and Evaluation

The study utilizes a dataset from Koko, an anonymous emotional support network, comprising approximately 106,000 binary-labeled posts. Unlike general social media platforms where crisis events are rare, Koko features a high "base rate" of crisis (roughly 20%), providing a rich environment for training. For evaluation, 200 crisis samples were manually annotated with "gold standard" human explanations.

Key Findings

LIME vs. Attention

A significant finding was that while attention weights improve detection performance, LIME proved superior for generating human-readable explanations. This suggests a divergence between the features a model uses for internal computation and the evidence required by a human to justify a label.

Attention heatmaps visualizing word importance for crisis detection.

The Value of Phrases

By using dependency parsing to extract subtrees, the model produces natural-sounding rationales (e.g., "to kill myself") rather than disjointed keywords. This linguistic coherence is vital for building trust with human-in-the-loop systems.

Performance Metrics

The neural models significantly outperformed the logistic baseline in detection, achieving an F1 score of 0.80 compared to 0.66. In terms of explanation quality, the Neural+LIME strategy led with a ROUGE-1 F1 score of 0.536, substantially higher than Neural+Attention (0.406).

Conclusion and Future Directions

While the Neural+LIME approach offers high-quality explanations, its computational cost remains a hurdle for real-time deployment, with single explanations taking up to a minute to generate. Future research may focus on "compression-based" models that are inherently concise or more efficient architectures. Ultimately, this work demonstrates that providing the "why" is just as important as the "what" in the sensitive domain of mental health crisis intervention.

Summary updated

BibTeX
@inproceedings{kshirsagar2017detecting,
  author = {Rohan Kshirsagar and Robert R. Morris and Sam Bowman},
  title = {{Detecting and Explaining Crisis}},
  booktitle = {CLPsych @ ACL},
  year = {2017},
  doi = {10.18653/v1/w17-3108},
}
Figure