Automated Utterance Labeling of Conversations Using Natural Language Processing

Maria Laricheva, Chiyu Zhang, Yan Liu, Guanyu Chen, Terence Tracey, Richard Young, Giuseppe Carenini

Abstract

In psychological research, analyzing conversational data through utterance labeling is a critical but resource-intensive task. This paper presents a specialized natural language processing (NLP) framework designed to automate this process. By utilizing a hierarchical labeling system and a domain-adapted transformer model, RoBERTa-CON, the researchers achieved near-human performance in binary classification tasks and established a robust baseline for multi-label behavioral analysis. This approach significantly reduces the manual labor required for behavioral coding while maintaining high levels of accuracy.

Introduction

Psychological researchers frequently rely on "utterance labeling"—the line-by-line analysis of conversations—to decode complex cognitive processes and emotional states. However, this methodology is notoriously labor-intensive, often requiring four to six hours of manual coding for every hour of recorded dialogue.

Automating this process using NLP is historically difficult due to three primary factors:

  1. Multilabel Complexity: Human speech is rarely singular in intent; a single sentence can simultaneously convey multiple emotions or behaviors.
  2. High Cardinality: Psychological coding schemes often involve dozens of specific labels, leading to significant data imbalance where rare behaviors are underrepresented.
  3. Data Scarcity: Privacy constraints and the high cost of expert annotation mean that high-quality training datasets are often much smaller than those used in standard NLP benchmarks.

Methodology

The study compared traditional machine learning models (Naive Bayes, Random Forest, SVM, and Logistic Regression) against advanced deep learning architectures. The researchers introduced several innovations to overcome the challenges of psychological data.

Hierarchical Labeling System

To manage the complexity of 59 distinct labels, the researchers implemented a hierarchical structure rather than a single massive classifier:

  • EMO-COG: A high-level binary classification between "Emotion" and "Cognition/Behavior."
  • Coarse-Grained (EMO-8/COG-8): Utterances are grouped into eight broad categories for each branch.
  • Fine-Grained (EMO-FULL/COG-FULL): The model attempts to predict the original, highly specific labels.

Domain Adaptation: RoBERTa-CON

The core of the framework is RoBERTa-CON. While standard RoBERTa models are trained on general text like Wikipedia, psychological dialogue is often repetitive, messy, and emotionally nuanced. The researchers performed "pre-pre-training" on a massive external database of over 1,500 therapy transcripts from the Alexander Street Press. This allowed the model to learn the specific linguistic patterns of psychological discourse before being fine-tuned on the study's primary dataset.

Contextual Awareness

Recognizing that the meaning of a sentence often depends on what was said previously, the model was designed to use contextual windows. To predict a label for a specific utterance ($U_i$), the model was provided with the current line plus the two preceding lines ($U_{i-2}, U_{i-1}$).

Experimental Results

The primary dataset consisted of 63 transcripts (7,965 utterances) focusing on youth transitioning to adulthood, featuring interactions between peers, parents, and siblings.

Performance Metrics

RoBERTa-CON emerged as the superior model across nearly all metrics:

  • Binary Classification (EMO-COG): The model achieved a 98.71% Weighted F1 score, effectively matching the reliability of human coders.
  • Coarse-Grained Classification: The model maintained solid performance with ~71% W-F1 for EMO-8 and ~68% for COG-8.
  • Fine-Grained Classification: As expected, performance decreased as granularity increased, with COG-FULL reaching ~45.8% W-F1. This drop highlights the ongoing challenge of the "long tail"—rare labels that appear infrequently in the training data.

Among non-deep learning methods, the Gradient Descent Support Vector Machine (GD-SVM) performed notably well, outperforming other conventional baselines.

Discussion and Key Insights

The study highlights that domain adaptation is the most significant factor in performance gains. By exposing the model to therapy-specific language before task-specific training, it developed a "vocabulary" better suited for emotional nuances. Furthermore, the hierarchical approach provides researchers with flexibility; if a study only requires distinguishing between emotional and cognitive statements, the model is nearly perfect.

However, the "data imbalance" problem remains a hurdle. Rare labels, such as "fear" (representing only 6% of the emotion data), showed significantly lower accuracy than dominant classes like "joy" or "agreement."

Conclusion

While the model is not yet a total replacement for human experts at the most granular level of coding, it represents a major step forward. RoBERTa-CON serves as a powerful tool for navigating and categorizing massive datasets, allowing researchers to focus their manual efforts on the most complex or rare utterances. Future research will explore semi-supervised and transfer learning techniques to further improve accuracy on minority classes.

Summary updated

BibTeX
@inproceedings{laricheva2022automated,
  author = {Maria Laricheva and Chiyu Zhang and Yan Liu and Guanyu Chen and Terence Tracey and Richard Young and Giuseppe Carenini},
  title = {{Automated Utterance Labeling of Conversations Using Natural Language Processing}},
  booktitle = {SBP-BRiMS},
  year = {2022},
  doi = {10.1007/978-3-031-17114-7\_23},
}
Figure