Abstract
This paper introduces a neural model that learns to represent a speaker’s unique style and intent dynamically as a conversation unfolds. By using an unsupervised objective to "reconstruct" utterances based on conversation history, the model creates evolving speaker embeddings that significantly improve performance on tasks like topic prediction in socialbots and dialog act classification. (Hao Cheng, Hao Fang, Mari Ostendorf. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.)
Introduction
In Natural Language Processing (NLP), context is paramount. While significant progress has been made in modeling what is being said, modeling who is saying it remains a challenge—especially in privacy-sensitive applications where metadata like age, gender, or user IDs are unavailable or unreliable (e.g., a shared smart speaker).
Existing methods often rely on static "snapshots" of a user or discrete metadata. However, people are dynamic; their interests and speaking styles can shift during a single interaction. The authors tackle the challenge of building a speaker model that relies solely on language (unsupervised), updates its understanding turn-by-turn, and captures human-interpretable "latent modes" of behavior.
Methods
The authors propose a Dynamic Speaker Model consisting of three interconnected neural components designed to track speaker state across a conversation.
- Latent Mode Analyzer: This component looks at a single utterance and compares it against a set of $K$ "global mode vectors" (learned clusters of speaking styles). It uses an attention mechanism to determine which "modes" (e.g., being terse, being talkative) the current turn matches.
- Speaker State Tracker: An LSTM-based module that maintains a "running summary" of the speaker across the conversation. At every turn, it takes the output of the Latent Mode Analyzer and updates the speaker's embedding.
- Speaker Language Predictor: This is the "engine" for unsupervised learning. It attempts to predict the words of the current turn, conditioned on the speaker's current state. By forcing the model to reconstruct the speaker's language, it learns to encode the speaker's specific linguistic regularities into the embedding.
Experimental Setup
The researchers evaluated their model on two distinct types of conversational data:
- User Topic Decision Dataset: A corpus of human-socialbot interactions from the 2017 Alexa Prize (Sounding Board system). It contains ~31,000 conversations where users accept or reject suggested topics.
- Switchboard Dialog Act Corpus (SwDA): A standard benchmark of human-human telephone conversations annotated with 42 "dialog act" labels (e.g., Statement, Question, Agreement).
- Fisher English Training Speech: A large unlabelled corpus used for unsupervised pre-training to boost the dialog act classification task.
Key Results and Insights
The dynamic model demonstrated that updating the speaker embedding at every turn performs significantly better than using a fixed profile. Language is a signal that gets clearer as the conversation progresses.
- Socialbot Topic Prediction: The dynamic model achieved 72.4% accuracy and a 20.0% reduction in normalized cross-entropy (N-CE), significantly outperforming TopicPrior baselines and standard auto-encoders.
- Dialog Act Classification: On the SwDA benchmark, the model achieved a state-of-the-art accuracy of 78.6% when pre-trained on the Fisher corpus.
- Style vs. Content: By using global mode vectors, the model effectively separates how someone speaks from what they are talking about. This allows the model to pick up on traits like "talkativeness" or "politeness" independently of the topic.
- Interpretable Latent Spaces: Even though training is unsupervised, the learned "modes" align with human concepts, such as distinguishing between male and female speaking patterns or specific backchanneling behaviors.
Conclusion
The Dynamic Speaker Model provides a robust framework for capturing evolving speaker characteristics without requiring explicit metadata. While the current model focuses on text, future work could integrate acoustic features (prosody, pitch) to further enrich speaker embeddings. Additionally, "guided" learning could nudge latent modes toward capturing specific traits like sentiment or personality.
Summary updated