Modeling Dialogue Structure with Adjacency Pair Analysis and Hidden Markov Models

Kristy Elizabeth Boyer, Robert Phillips, Eun Young Ha, Michael Wallis, Mladen Vouk, James LesterNAACL 2009

TL;DR

This paper proposes a novel method for automatically identifying the underlying structure of tutorial dialogues by combining adjacency pair analysis with Hidden Markov Models (HMMs). By treating dialogue act pairs as atomic units, they create more descriptive models of tutoring strategies, offering a data-driven approach to understanding and designing intelligent tutoring systems.

What Problem Does This Paper Address?

Understanding the structure of human-human dialogue, particularly in educational settings like tutoring, is crucial for developing more effective intelligent tutoring systems (ITS). Traditional approaches often rely on manual annotation or top-down rule-based systems. The authors highlight a gap in automatic, data-driven methods for identifying "dialogue modes" or tutoring strategies that are implicitly present in the conversation. They argue that empirical identification of these strategies can inform ITS design and our understanding of the learning process itself.

Methods

The core of their methodology involves two main steps:

  1. Adjacency Pair Analysis: They first manually annotate a corpus of tutorial dialogues with dialogue acts (e.g., Question, Statement, Positive Feedback). Then, they use a chi-squared (χ²) test to empirically identify statistically significant "adjacency pairs" of dialogue acts (e.g., a tutor's question followed by a student's statement). These identified pairs are treated as single, atomic units.
  2. Hidden Markov Models (HMMs): They then train an HMM on sequences of these atomic units (which include both individual dialogue acts not part of a significant pair and the identified adjacency pairs). The HMM learns to infer underlying "dialogue modes" (hidden states) that best explain the observed sequences of dialogue acts and pairs. Each hidden state is characterized by a probability distribution over the observed symbols, effectively defining a tutoring strategy.

This approach differs from standard HMMs applied to individual dialogue acts by incorporating the relational information captured by adjacency pairs, leading to a more descriptive model.

Datasets

The study uses a corpus of human-human tutorial dialogues collected in an introductory computer science domain. This corpus consists of 4,864 dialogue moves from 43 learners interacting remotely with a tutor. The dialogues were manually tagged with dialogue acts. No new benchmark datasets were introduced; the focus was on analyzing an existing corpus.

Key Insights and Ideas

  • Dialogue Acts as Building Blocks: Dialogue acts are fundamental units, but their sequential relationships (adjacency pairs) carry significant structural information about the interaction.
  • Empirical Identification of Strategies: Instead of pre-defining tutoring strategies, this method allows them to be discovered empirically from the data.
  • HMMs for Dialogue Modes: HMMs are well-suited for modeling sequential data and can effectively capture the probabilistic transitions between different tutoring strategies (dialogue modes).
  • Adjacency Pairs Enhance Models: Treating significant adjacency pairs as single units leads to more descriptive HMMs compared to using only individual dialogue acts. This is because it captures the inherent "turn-taking" or "response" structure that is common in dialogues.
  • Interpretable Dialogue Modes: The learned hidden states (dialogue modes) can be interpreted in terms of common tutoring behaviors, such as "Tutor Lecture," "Tutor Evaluation," "Extra-Domain & Grounding," and "Student Questions & Reflection."

Key Results

  • The best-fit HMM model was found to have four hidden states, representing distinct dialogue modes.
  • State 0 ("Tutor Lecture") was the most prevalent, accounting for 63% of the corpus, and was characterized by tutor statements and positive feedback. This state was also highly stable, with a self-transition probability of 0.835.
  • State 3 ("Student Questions & Reflection") was identified as an interactive state where students ask questions and provide statements.
  • A significant transition was observed from State 3 to State 0 (nearly 60% of the time), suggesting that after a student-driven phase, the tutor might shift to a more directive teaching mode.
  • The paper demonstrates that this approach can generate a descriptive model of dialogue structure without significantly increasing model complexity.

Limitations and Future Work

  • Model Comparison: The authors plan to compare their HMM approach with other types of graphical models in future work.
  • Correlating with Outcomes: A key future direction is to correlate the identified dialogue profiles (from the HMM) with actual learning and affective outcomes of the tutoring sessions. This would directly inform ITS design.
  • Task State and Deeper Content: They suggest leveraging knowledge of the task state and surface-level utterance content (beyond dialogue acts) to further refine the descriptive and predictive power of their models.
  • Initial Maximum States: The choice of n=15 as an initial maximum number of states was somewhat arbitrary, though justified by exceeding hypothesized ranges and the AIC measure.

Summary updated

BibTeX
@inproceedings{boyer2009modeling,
  author = {Kristy Elizabeth Boyer and Robert Phillips and Eun Young Ha and Michael Wallis and Mladen Vouk and James Lester},
  title = {{Modeling Dialogue Structure with Adjacency Pair Analysis and Hidden Markov Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure