Identifying Small Talk in Natural Conversations

Steffen Frenzel, Annette Hautli-JaniszNAACL 2025

TL;DR

This paper tackles the challenge of automatically identifying "small talk" in natural conversations, a crucial but often overlooked aspect of human interaction. They introduce a novel, theoretically-grounded dataset of German public service encounters and demonstrate that traditional statistical models can achieve performance comparable to state-of-the-art large language models (LLMs) for this task, highlighting the importance of context and careful annotation.

What Problem Does This Paper Address?

Small talk, the seemingly trivial conversational filler, plays a vital role in building social connections, conveying values, and establishing rapport. Despite its ubiquity, precisely defining and automatically identifying small talk is a significant challenge. This is due to its highly context-dependent nature, requiring an understanding of interlocutors' backgrounds and situational contexts. Furthermore, collecting and transcribing natural conversations for training data is labor-intensive and expensive. This paper addresses the lack of robust methods and annotated datasets for small talk detection, particularly in German, and investigates how well different computational models can perform this task.

Methods

The authors developed a theoretically-motivated annotation scheme for small talk, focusing on its function in maintaining social relations and creating a basis for the main discussion. They conducted manual annotation rounds, refining the guidelines until achieving moderate inter-annotator agreement (0.534 Cohen's Kappa). For automatic classification, they employed four models:

  • Logistic Regression (LR): Used with both TF-IDF and sentence embeddings (distilbert).
  • Support Vector Machine (SVM): Also trained with TF-IDF and sentence embeddings, with hyperparameter tuning.
  • GBERT: A BERT model specifically trained on German, with careful fine-tuning to mitigate overfitting.
  • GPT-4.0: Evaluated using zero-shot, few-shot, and chain-of-thought prompting strategies.

The core of their approach lies in comparing these diverse models on a newly created dataset, aiming to understand the trade-offs between model complexity and performance.

Datasets

The primary dataset used is the PSE v1.0 dataset (Espinoza et al., 2024). This dataset comprises over 106 German public service encounters, totaling more than 31,000 speaker turns and 433,780 tokens. These are verbatim transcripts of conversations between citizens and state representatives, often involving sensitive personal topics. The authors created a new, human-annotated subset of this data specifically for small talk detection, containing over 2,600 utterances.

Key Insights and Ideas

  • Context is King (Even for Small Talk): The error analysis revealed that small talk often occurs in extended sections of a conversation rather than isolated utterances. This suggests that models need to consider conversational context, not just individual turns, for accurate detection.
  • Traditional Models Can Compete: Surprisingly, simpler models like Logistic Regression and SVM, when properly fine-tuned and combined with effective sentence embeddings, can achieve performance comparable to more complex LLMs like GPT-4.0 for this task. This challenges the assumption that only the largest models can handle nuanced linguistic tasks.
  • Annotation Challenges Persist: Even with detailed guidelines, human annotators struggled with identifying the "transition zones" between small talk and task-oriented conversation, indicating the inherent ambiguity of the phenomenon.
  • Imbalanced Data is a Hurdle: The dataset is highly imbalanced (more non-small talk than small talk), posing a significant challenge for model training and often leading to bias towards the majority class.

Key Results

The study found that while LLMs like GPT-4.0 can achieve good results with few-shot prompting (F1-score of 0.68), traditional models also performed well. Specifically:

  • Logistic Regression with distilbert embeddings achieved an F1-score of 0.73.
  • SVM with distilbert embeddings achieved an F1-score of 0.68.
  • GPT-4.0 (few-shot) achieved an F1-score of 0.68.

Interestingly, the GBERT model, despite being a specialized German language model, performed the worst, likely due to the small training dataset size and class imbalance. The error analysis highlighted that models often misclassify standalone utterances as small talk when surrounding utterances are not, suggesting a reliance on lexical rather than contextual cues.

Limitations and Future Work

The authors acknowledge that operationalizing the concept of small talk remains the biggest challenge. They learned that conversational context is crucial for human annotators, and this kind of information needs to be better integrated into automatic classification models, perhaps through additional engineered features. Future work could focus on developing more sophisticated methods for incorporating conversational context into models and exploring techniques to better handle the class imbalance inherent in small talk detection datasets.

Summary updated

BibTeX
@inproceedings{frenzel2025identifying,
  author = {Steffen Frenzel and Annette Hautli-Janisz},
  title = {{Identifying Small Talk in Natural Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}

Loading related content...

Figure