Dialog Context Language Modeling with Recurrent Neural Networks

Bing Liu, Ian R. Lane

Abstract

In this work, we propose contextual language models that incorporate dialog-level discourse information into language modeling. Previous works on contextual language models treat preceding utterances as a sequence of inputs without considering dialog interactions. We design recurrent neural network (RNN) based contextual language models that specifically track the interactions between speakers in a dialog. Experimental results on the Switchboard Dialog Act Corpus show that the proposed model outperforms conventional single-turn based RNN language models by 3.3% on perplexity. The proposed models also demonstrate advantageous performance over other competitive contextual language models.

The Challenge of Conversational Context

Traditional language models (LMs) predict the next word based on a local window of text. While "contextual" LMs exist, they typically treat preceding text as a flat sequence, much like a single-author document. However, dialog is fundamentally different. It involves multiple participants, turn-taking, and "grounding"—where speakers acknowledge and build upon each other's points.

A speaker’s current sentence is often a direct continuation of their own previous statement, even if interrupted by another person’s response. Standard models struggle to capture these multi-party dynamics, leading to higher uncertainty (perplexity) when predicting conversational flow.

Modeling Speaker Interactions

To address these challenges, the researchers developed two primary architectures that explicitly model the relationship between turns:

  1. Interactive Dialog Context LM (IDCLM): This model creates a "cross-stitch" of information. When modeling Speaker A’s current turn, the model connects the final state of Speaker A’s previous turn directly to the start of the current turn. Meanwhile, Speaker B’s intervening turn is used as a context vector that influences every word prediction, allowing the model to track an individual thread while remaining aware of the partner's input.
  2. External State IDCLM (ESIDCLM): This architecture introduces a separate, high-level RNN that specifically tracks the state of the entire dialog. While the word-level RNN handles the text, this external RNN updates a "dialog state" after every turn, acting as a summary of the conversation's progress.
The ESIDCLM architecture utilizes a hidden dialogue state to track interactions across multiple turns and speakers.

Both models utilize Long Short-Term Memory (LSTM) units to handle long-range dependencies and are initialized with pre-trained word embeddings to enhance performance.

Experimental Results

The models were evaluated on the Switchboard Dialog Act Corpus (SwDA), utilizing approximately 98.7K turns for training. The ESIDCLM model achieved a perplexity of 58.4, compared to 60.4 for a standard single-turn RNNLM—a relative improvement of roughly 3.3%.

Comparison of perplexity results across various models and context lengths (K). The proposed models (IDCLM, ESIDCLM, DACLM) consistently outperform standard baselines.

As shown in the results, the proposed models also outperformed other popular contextual methods, such as Bag-of-Words (BoW) context and standard Document-level RNNs (DRNNLM).

Key Insights and Analysis

  • Speaker Continuity: Linking a speaker's current turn to their own previous turn proved highly effective, acknowledging the consistent internal logic that persists across a speaker's contributions.
  • Reference Tracking: Analysis by Parts of Speech (POS) revealed a massive 16.8% improvement in predicting personal pronouns. This suggests the models are significantly better at resolving who or what a speaker is referring to based on previous context.
  • Implicit State Learning: The proposed models performed nearly as well as a model provided with "perfect" human-labeled dialog act tags. This indicates that the RNNs successfully learn to represent the state of the conversation (e.g., identifying when a question has been asked) implicitly from raw text.

Conclusion

By explicitly accounting for the unique structure of human conversation, these contextual RNN models provide a more robust framework for language modeling in dialog. While the models excel at tracking flow and references, future work may explore specialized mechanisms to better capture topical content and improve the prediction of specific nouns.

Summary updated

Abstract

In this work, we propose contextual language models that incorporate dialog level discourse information into language modeling. Previous works on contextual language model treat preceding utterances as a sequence of inputs, without considering dialog interactions. We design recurrent neural network (RNN) based contextual language models that specially track the interactions between speakers in a dialog. Experiment results on Switchboard Dialog Act Corpus show that the proposed model outperforms conventional single turn based RNN language model by 3.3% on perplexity. The proposed models also demonstrate advantageous performance over other competitive contextual language models.
BibTeX
@inproceedings{liu2017dialog,
  author = {Bing Liu and Ian R. Lane},
  title = {{Dialog Context Language Modeling with Recurrent Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=bHbhbaM-fDo},
}
Figure