Logic-driven Indirect Supervision: An Application to Crisis Counseling

Mattia Medina Grespan, Meghan Broadbent, Xinyao Zhang, Katherine Axford, Brent Kious, Zac E. Imel, Vivek Srikumar

Abstract

Identifying high-risk utterances in crisis counseling is critical for real-time intervention but suffers from a lack of expert-labeled data. This paper introduces a logic-driven indirect supervision framework that leverages abundant, coarse session-level risk labels to improve fine-grained utterance-level classification. By encoding clinical domain knowledge into differentiable logic rules using R-product t-norm relaxations, the model achieves a 3.5% F1-score improvement and a significant boost in recall for high-risk messages compared to standard multi-task learning baselines.

Introduction

In text-based crisis counseling, such as the 988 Lifeline, real-time identification of high-risk utterances is essential for supervisor prioritization and counselor feedback. However, training models for this task faces a significant "data bottleneck." Labeling individual messages for specific risk factors (e.g., "imminent risk" vs. "passive ideation") requires expert clinical knowledge and is prohibitively expensive. Conversely, session-level labels—where counselors tag an entire encounter as "high" or "low" risk—are abundant but coarse and noisy. This work explores how to bridge this granularity gap by using session-level signals to supervise fine-grained utterance classifiers.

Example of a crisis chat transcript with expert-labeled risk codes (D-codes).

Methodology

The proposed framework utilizes Logic-driven Indirect Supervision. Clinical guidelines are treated as logical constraints that link session-level and utterance-level data.

  1. Architecture: The model uses a shared RoBERTa-based backbone. It processes a chat session to generate message embeddings, followed by a Transformer encoder to capture conversational context. The model outputs both a session-level risk label and multi-label risk classifications for each utterance.
  2. Logical Constraints: Structural dependencies are defined using predicate logic. For instance:
  3. Joint Constraint: A session labeled "Lower Risk" cannot contain utterances with "High Risk" codes (e.g., "Attempt in Progress").
  4. D-Code Dependencies: Specific codes imply others; e.g., a "Prior Attempt" logically implies "Lifetime Ideation."
  5. Differentiable Loss: To integrate these rules into neural network training, the authors employ R-product t-norm relaxations. This converts discrete logic into a continuous loss function, penalizing the model when its predictions violate clinical consistency.
Overview of the logic-driven indirect supervision framework, integrating counselor session labels and expert utterance labels via structural dependencies.

Data and Experimental Setup

The study utilizes two disjoint datasets from SafeUT, a regional suicide crisis hotline:

  • Dataset U (Utterance-level): 425 sessions (~4,912 client utterances) labeled post-hoc by experts using a 9-dimensional risk coding scheme (D-codes).
  • Dataset E (Encounter-level): 5,990 sessions labeled by the original counselors as simply "Higher" or "Lower" risk.

Crucially, no session exists in both sets, forcing the model to learn the relationship between the two levels of abstraction indirectly through the logical constraints.

Results and Discussion

The logic-driven model (MT+Rules) significantly outperformed baselines:

  • Performance Boost: It achieved a 50.0 F1-score on utterance classification, representing a 3.5% improvement over multi-task learning and a 6.8% improvement over single-task models.
  • Clinical Safety: Most notably, recall for high-risk sessions jumped from 24.8% (baseline) to 84.4%. In clinical triage, prioritizing recall is vital to ensure no high-risk cues are missed.
  • Consistency: The model demonstrated higher logical robustness, violating clinical constraints far less frequently than purely data-driven approaches, making the output more trustworthy for human supervisors.

Conclusion

This research demonstrates that domain knowledge can effectively bridge the gap between coarse and fine-grained data. By enforcing logical consistency, models can learn more reliable representations from noisy, abundant session data. While the model is sensitive to hyper-parameter tuning for logic weights, its ability to prioritize recall makes it a promising tool for "human-in-the-loop" assistant systems in crisis counseling. Future work includes exploring larger language models and prompt-based architectures to further enhance performance.

Summary updated

Abstract

Ensuring the effectiveness of text-based crisis counseling requires observing ongoing conversations and providing feedback, both labor-intensive tasks. Automatic analysis of conversations—at the full chat and utterance levels—may help support counselors and provide better care. While some session-level training data (e.g., rating of patient risk) is often available from counselors, labeling utterances requires expensive post hoc annotation. But the latter can not only provide insights about conversation dynamics, but can also serve to support quality assurance efforts for counselors. In this paper, we examine if inexpensive—and potentially noisy—session-level annotation can help improve label utterances. To this end, we propose a logic-based indirect supervision approach that exploits declaratively stated structural dependencies between both levels of annotation to improve utterance modeling. We show that adding these rules gives an improvement of 3.5% f-score over a strong multi-task baseline for utterance-level predictions. We demonstrate via ablation studies how indirect supervision via logic rules also improves the consistency and robustness of the system.
BibTeX
@inproceedings{grespan2023logic,
  author = {Mattia Medina Grespan and Meghan Broadbent and Xinyao Zhang and Katherine Axford and Brent Kious and Zac E. Imel and Vivek Srikumar},
  title = {{Logic-driven Indirect Supervision: An Application to Crisis Counseling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.654},
}
Figure