Dialog context language modeling with recurrent neural networks

Bing Liu, Ian R. Lane

TL;DR

Researchers from Carnegie Mellon University have developed a new way to improve language modeling for conversations by explicitly tracking interactions between speakers. By using specialized Recurrent Neural Network (RNN) architectures that account for turn-taking, their models achieved a 3.3% improvement in perplexity over standard models on the Switchboard Dialog Act Corpus.

What Problem Does This Paper Address?

Traditional language models (LMs) predict the next word based on a local window of text. While "contextual" LMs exist, they typically treat preceding text as a flat sequence, much like a single-author document or a book chapter.

However, dialog is different. It involves multiple participants, turn-taking, and "grounding" (where speakers acknowledge each other's points). A speaker’s current sentence is often a direct continuation of their own previous statement, interrupted by the other person’s response. Standard models struggle to capture these multi-party dynamics, leading to higher uncertainty (perplexity) when predicting what a speaker will say next in a conversation.

Methods

The authors propose two primary architectures to better model these interactions:

  1. Interactive Dialog Context LM (IDCLM): Instead of just feeding the previous turn into the model, IDCLM creates a "cross-stitch" of information. When modeling Speaker A’s current turn (Turn $k$), the model connects the final state of Speaker A’s previous turn (Turn $k-2$) directly to the start of the current turn. Meanwhile, Speaker B’s intervening turn (Turn $k-1$) is used as a "context vector" that influences every word prediction in the current turn. This allows the model to track a speaker's individual thread while remaining aware of the partner's input.

  2. External State IDCLM (ESIDCLM): This version introduces a separate, "high-level" RNN that specifically tracks the state of the entire dialog. While the word-level RNN handles the actual text, this external RNN updates its "dialog state" after every turn. This state then serves as the context for the next turn, effectively acting as a summary of the conversation's progress so far.

Both models use Long Short-Term Memory (LSTM) units to handle long-range dependencies and are initialized with pre-trained word embeddings from Google News.

Datasets

The models were evaluated on the Switchboard Dialog Act Corpus (SwDA).

  • Scale: Approximately 98.7K turns for training.
  • Features: This dataset is unique because it includes human-labeled "dialog acts" (e.g., "Statement," "Acknowledge," "Question"), which the authors used to create a "gold standard" baseline for comparison.

Key Insights and Ideas

  • Speaker Continuity Matters: One of the most effective ideas was linking a speaker's current turn to their own previous turn ($k \to k-2$). This acknowledges that people often have a consistent internal logic or "train of thought" that persists across turns.
  • Context Helps Most with References: By analyzing performance based on Parts of Speech (POS), the authors found the biggest gains in pronouns. This makes sense: to know who "he" or "it" refers to, the model must have a good representation of the previous context.
  • Implicit vs. Explicit Modeling: The proposed models performed nearly as well as a model provided with "perfect" human-labeled dialog act tags. This suggests that the RNNs are successfully learning to represent the "state" of the conversation (e.g., realizing that a question was just asked) implicitly from the raw text.

Key Results

  • Perplexity Reduction: The ESIDCLM model achieved a perplexity of 58.4, compared to 60.4 for a standard single-turn RNNLM—a relative improvement of roughly 3.3%.
  • Beating Baselines: The proposed models outperformed other popular contextual methods, such as Bag-of-Words (BoW) context and standard Document-level RNNs (DRNNLM).
  • POS Gains: The models showed a massive 16.8% improvement in predicting personal pronouns (PRP) compared to the baseline, highlighting their ability to track entities across turns.

Limitations and Future Work

  • Parameter Overhead: The ESIDCLM model requires more parameters due to the secondary "dialog-tracking" RNN. The authors note this can make the model more prone to overfitting if the training data is limited.
  • Noun Prediction: Interestingly, the models were less effective at improving the prediction of nouns. The authors suggest that while the models are great at tracking the flow of a conversation, they might need more specialized mechanisms to capture specific topical content or "entities."
  • Single Dataset: The study focused entirely on the Switchboard corpus; testing on more diverse or modern datasets (like social media threads or technical support chats) would be a logical next step.

Summary updated

Abstract

In this work, we propose contextual language models that incorporate dialog level discourse information into language modeling. Previous works on contextual language model treat preceding utterances as a sequence of inputs, without considering dialog interactions. We design recurrent neural network (RNN) based contextual language models that specially track the interactions between speakers in a dialog. Experiment results on Switchboard Dialog Act Corpus show that the proposed model outperforms conventional single turn based RNN language model by 3.3% on perplexity. The proposed models also demonstrate advantageous performance over other competitive contextual language models.
BibTeX
@inproceedings{liu2017dialog,
  author = {Bing Liu and Ian R. Lane},
  title = {{Dialog context language modeling with recurrent neural networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=bHbhbaM-fDo},
}

Loading related content...

Figure