What Motivates You? Benchmarking Automatic Detection of Basic Needs from Short Posts

Sanja Stajner, Seren Yenikent, Bilal Ghanem, Marc Franco-SalvadorACL 2021

TL;DR

This paper tackles the challenge of automatically detecting basic psychological needs (autonomy, competence, relatedness) from short social media posts. By benchmarking various NLP models, they demonstrate that advanced architectures like BERT can achieve performance comparable to human annotators, especially when framing the problem as multiple binary classification tasks rather than a single multi-class one.

What Problem Does This Paper Address?

Understanding human motivation is crucial across various fields, from education to organizational psychology. The Self-Determination Theory (SDT) posits that three fundamental psychological needs—autonomy (control over one's actions), competence (feeling effective), and relatedness (feeling connected to others)—drive behavior and well-being. Traditionally, these needs are assessed through self-report questionnaires, which can be subject to biases like social desirability. While implicit methods exist, they are often time-consuming. This paper addresses the gap of automatically detecting these basic needs from short, informal text, specifically social media posts, which are abundant and reflect everyday thoughts and feelings. The challenge lies in the brevity and often implicit nature of these expressions, making it difficult for machines to accurately infer underlying motivational states.

Methods

The authors frame the detection of basic needs as a classification problem. They explore two main approaches:

  • Ternary Classification: A single model predicts one of the three basic needs (autonomy, competence, or relatedness) for each post.
  • Binary Classification: Three separate models are trained, each predicting whether a specific need (autonomy, competence, or relatedness) is present or absent in a post.

They benchmark several NLP architectures, including:

  • BOW (Bag-of-Words): A traditional feature-based model using TF-IDF weighting with a Support Vector Machine (SVM).
  • Char-CNN: A Convolutional Neural Network that operates on character sequences, using character embeddings.
  • BiLSTM: A Bidirectional Long Short-Term Memory network that uses FastText word embeddings and an attention mechanism.
  • BERT and BERT+BiLSTM: State-of-the-art transformer-based models. BERT is fine-tuned, and BERT+BiLSTM further processes BERT's output with a BiLSTM.

The study also includes a human performance evaluation by a psychologist to establish a performance ceiling.

Datasets

The primary dataset used is the Basic Psychological Needs Corpus, specifically its first two layers of annotation. This corpus, publicly available, contains Twitter posts annotated for basic needs. The dataset was pre-filtered for non-emotional posts and those lacking sufficient signal. It comprises 6334 posts, with distributions of 1229 for competence, 1771 for autonomy, and 3334 for relatedness. The authors also created a subset of 40 instances per class for direct comparison with a trained human annotator.

Key Insights and Ideas

  • Hashtags Matter: The presence of hashtags significantly improves model performance, suggesting they carry valuable psycholinguistic signals related to basic needs (e.g., #proud for competence, #relationship for relatedness).
  • Binary vs. Ternary: Framing the problem as three independent binary classification tasks (presence/absence of each need) is more effective and applicable than a single ternary classification task. This allows for posts to be labeled with none, one, or multiple needs, better reflecting the nuanced reality of human motivation.
  • BERT's Power: Advanced transformer models like BERT and BERT+BiLSTM significantly outperform traditional methods, demonstrating their ability to capture complex linguistic patterns.
  • Autonomy vs. Competence Confusion: Misclassifications were most common between autonomy and competence. This is attributed to their shared "self-originated" nature in SDT, leading to potential topic overlaps, whereas relatedness is more easily distinguished due to its focus on others.
  • Human-Level Performance: The best-performing model (BERT+BiLSTM) achieved performance comparable to a trained human annotator on a subset of the data.
  • Intensity Signals: The class probabilities from the binary models show a strong correlation with human-perceived intensity levels of basic needs (low, high), suggesting potential for more granular analysis.

Key Results

  • Ternary Classification: BERT+BiLSTM achieved the highest F1-scores, with macro-averaged F1 scores of 0.73 (without hashtags) and 0.76 (with hashtags). The trained human annotator achieved an F1-score of 0.77 on a subset.
  • Binary Classification: On the dataset with hashtags, BERT+BiLSTM achieved F1-scores of 0.74 for autonomy, 0.79 for competence, and 0.88 for relatedness. Without hashtags, these were 0.69, 0.74, and 0.84, respectively.
  • Human Annotation Comparison: The binary classification approach showed high agreement with human annotations, particularly in identifying posts with no need signal and in capturing multiple needs. The probabilities from binary models correlated well with human judgments of need intensity.

Limitations and Future Work

  • Dataset Limitations: The dataset was pre-filtered for emotional and needs signals, limiting its applicability to general social media text. Demographic information was not available, preventing analysis of model performance across different groups.
  • Generalizability: The models were trained and evaluated on Twitter data. Their performance on other text types (e.g., blogs, forums) or in different contexts is uncertain.
  • Failure Modes: Misclassifications were primarily observed between autonomy and competence. The study did not thoroughly investigate performance on non-Twitter data.
  • Future Work: The authors suggest exploring models that can handle more general text, potentially incorporating demographic information, and further investigating the relationship between model probabilities and perceived need intensity. They also highlight the potential for these models to aid in understanding user needs in organizational and educational contexts, provided careful bias control and representative data sampling. The potential for misuse in decision-making processes (e.g., hiring) is also noted, emphasizing the need for ethical considerations.

Summary updated

BibTeX
@inproceedings{stajner2021what,
  author = {Sanja Stajner and Seren Yenikent and Bilal Ghanem and Marc Franco-Salvador},
  title = {{What Motivates You? Benchmarking Automatic Detection of Basic Needs from Short Posts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure