Attention-Based LSTM for Psychological Stress Detection from Spoken Language Using Distant Supervision

Genta Indra Winata, Onno Pepijn Kampman, Pascale Fung

TL;DR

Detecting psychological stress from spoken language is challenging due to the scarcity of labeled data. This paper demonstrates that an attention-based Bidirectional LSTM (BLSTM) model, pre-trained on "noisy" Twitter data labeled via hashtags (distant supervision) and then fine-tuned on interview transcripts, can effectively identify stress-related linguistic patterns with over 74% accuracy.

What Problem Does This Paper Address?

Psychological stress is a precursor to serious health issues like heart disease and depression, yet many people lack adequate emotional support. While linguistic studies show that our choice of words can signal stress levels, building automated detection systems for spoken language is difficult.

The primary hurdle is data: high-quality datasets of transcribed interviews are often too small to train deep learning models effectively. Conversely, social media data is abundant but "noisy" and differs significantly from spoken language. The authors tackle this by finding a way to leverage massive amounts of social media text to improve stress detection in the more specialized domain of interview transcripts.

Methods

The researchers explored several architectures, focusing on Long Short-Term Memory (LSTM) networks, which are well-suited for capturing long-term dependencies in sequences like sentences.

  1. Attention Mechanism: The core innovation is an attention layer added to the LSTM. Instead of treating every word in a sentence as equally important, the attention mechanism assigns "weights" to specific words. This allows the model to focus on highly informative terms (like "anxious" or "pressure") while ignoring filler words.
  2. Bidirectional LSTM (BLSTM): They used a bidirectional approach, processing the word sequence in both forward and backward directions to capture fuller context.
  3. Distant Supervision: To overcome the small size of their primary dataset, they used "distant supervision." They automatically labeled hundreds of thousands of tweets based on specific hashtags (e.g., #stressed vs. #relaxed) to pre-train the model's parameters before fine-tuning it on the high-quality interview data.
Fig. 2. Attention-based BLSTM architecture

Datasets

  • Natural Stress Emotion Corpus: A small but high-quality dataset consisting of transcripts from 38 students answering 12 progressively stress-provoking interview questions. It contains 2,272 utterances.
  • Stress Twitter Corpus: A massive "noisy" dataset of 367,312 tweets. The authors used a set of 42 "stressed" hashtags and 14 "unstressed" hashtags to automatically label the data for pre-training.

Key Insights and Ideas

  • Twitter as a "Warm-up": Pre-training on Twitter data significantly expanded the model's vocabulary and improved its robustness. Even though tweets are grammatically different from spoken transcripts, the underlying "stress vocabulary" overlaps enough to provide a major performance boost.
  • Visualizing "Focus": The attention mechanism acts as a window into the model's "thought process." By extracting attention weights, the authors created heatmaps showing that the model successfully identifies stressors like "employment," "graduation," and "difficult."
  • The Negation Challenge: A fascinating finding was the model's struggle with negation. For example, the phrase "I am not stressed" was often classified as "stressed" because the model focused heavily on the keyword "stressed" but failed to grasp the semantic impact of the word "not." This highlights a limitation in how these models learn complex grammatical structures from noisy data.
Fig. 3. Heatmap of attention layer weights for stressed utter- ances.

Key Results

The BLSTM with attention combined with distant supervision fine-tuning emerged as the clear winner:

  • Top Performance: Achieved 74.1% accuracy and a 74.3% F-score.
  • The "Fine-Tuning" Boost: Using distant supervision from Twitter improved the model's performance by 1.6% in accuracy and 2.1% in F-score compared to training on interview data alone.
  • Attention Matters: Models with the attention mechanism consistently outperformed their standard counterparts, proving that "selective listening" is key to detecting emotional states.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Context and Prosody: Currently, the model only looks at text. A more complete system would include acoustic features (tone of voice, pitch, pauses) and conversational context.
  • Grammar and Negation: The model needs more grammatically correct training data to better understand nuances like negation ("not stressed").
  • Real-world Application: The ultimate goal is to integrate this model into a "virtual therapist" platform that can recognize user stress in real-time and provide appropriate management advice or exercises.

Summary updated

Abstract

We propose a Long Short-Term Memory (LSTM) with attention mechanism to classify psychological stress from self-conducted interview transcriptions. We apply distant supervision by automatically labeling tweets based on their hashtag content, which complements and expands the size of our corpus. This additional data is used to initialize the model parameters, and which it is fine-tuned using the interview data. This improves the model's robustness, especially by expanding the vocabulary size. The bidirectional LSTM model with attention is found to be the best model in terms of accuracy (74.1%) and f-score (74.3%). Furthermore, we show that distant supervision fine-tuning enhances the model's performance by 1.6% accuracy and 2.1% f-score. The attention mechanism helps the model to select informative words.
BibTeX
@inproceedings{winata2018attention,
  author = {Genta Indra Winata and Onno Pepijn Kampman and Pascale Fung},
  title = {{Attention-Based LSTM for Psychological Stress Detection from Spoken Language Using Distant Supervision}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2018.8461990},
}
Figure