What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition

Piotr Zelasko, Raghavendra Pappagari, Najim Dehak

TL;DR

This paper investigates how transformer models can be used for dialog act recognition, a crucial task for understanding conversations. The authors find that providing broader conversational context and preserving punctuation significantly improves performance, while the specificity of dialog act labels has less impact.

What Problem Does This Paper Address?

Understanding human conversations is a complex challenge for AI. A key step is "dialog act recognition," which involves breaking down a conversation into meaningful units (dialog acts) that represent specific communicative functions, like asking a question or making a statement. This is more granular than just segmenting by speaker turns. Accurately identifying these dialog acts and their boundaries is essential for downstream applications like voice assistants and analyzing conversational flow. While transformer models have excelled in many NLP tasks, their application to dialog act recognition, especially with long conversational contexts, has been limited due to computational constraints. This paper aims to understand what factors are most important for transformers to effectively perform this task.

Methods

The researchers leverage two transformer architectures designed for long sequences: XLNet and Longformer.

  • XLNet, based on Transformer-XL, processes sequences in windows and propagates intermediate layer activations between windows to maintain context.
  • Longformer uses a modified self-attention mechanism with local sliding windows and global attention to reduce computational complexity, allowing it to process longer sequences more efficiently.

They adapt these models for dialog act recognition by adding a token classification layer and fine-tuning them. Crucially, they experiment with different configurations:

  • Turn-level models: Process each speaker turn in isolation, serving as a baseline.
  • Broad context models: Process entire conversations or large chunks, using techniques like concatenating turns with a special TURN token to provide context.

They also investigate the impact of punctuation and capitalization by comparing models trained on original transcripts versus those with all text converted to lowercase and punctuation removed.

Datasets

The study utilizes two well-established datasets for dialog act recognition:

  • Switchboard Dialog Act (SWDA): A corpus of telephone conversations, annotated with dialog acts.
  • Meeting Recorder Dialog Act (MRDA): A corpus of multi-speaker meetings, also annotated with dialog acts.

The paper notes that SWDA uses a 42-label set for dialog acts, while MRDA offers several label set granularities: a basic set of 5 labels, a general set of 12, and a full set of 51.

Key Insights and Ideas

  • Context is King (for Disambiguation): Providing broader conversational context significantly helps disambiguate dialog acts, especially those that are infrequent in the training data. This is a major win for understanding nuanced communication.
  • Punctuation is Powerful: The presence of punctuation and capitalization has a surprisingly large impact on model performance. Without them, models tend to under-segment conversations, merging disfluencies with neighboring dialog acts.
  • Label Specificity Matters Less for Segmentation: The granularity of the dialog act label set (e.g., 5 vs. 51 labels) has a minimal effect on the segmentation quality. This suggests that for tasks focused on identifying conversational structure, a simpler set of labels might suffice, easing annotation efforts.
  • Transformers Outperform Baselines: Even without context, transformer models like XLNet and Longformer significantly outperform traditional BiGRU models for dialog act recognition.
  • Longformer's Speed Advantage: Longformer offers a substantial speedup in inference time compared to XLNet, making it more practical for real-world applications, even if its recognition performance is slightly lower.

Key Results

  • Strong Segmentation Performance: The best models achieved Dialog Act Segmentation Error Rates (DSER) of 8.4% on SWDA and 14.2% on MRDA with original transcripts.
  • Contextual Improvement: Broad context models (XLNet+prop) outperformed turn-level models across most metrics. For example, in SWDA, the number of dialog acts never correctly recognized dropped from over 50% to less than 10% with context.
  • Punctuation's Impact: Removing punctuation led to noticeable drops in performance, particularly in segmentation quality. Models without punctuation tended to under-segment, merging disfluencies.
  • Speed: Longformer was approximately five times faster than XLNet for inference on a CPU.

Limitations and Future Work

  • Punctuation Dependency: The models' strong reliance on punctuation suggests that systems processing raw ASR output (which often lacks punctuation) will face challenges. The authors suggest exploring whether automatically retrieved punctuation can bridge this gap.
  • Longformer's Fixed Context: While Longformer handles long sequences, its local attention is limited to a fixed window size (512 tokens in this study), and its positional embeddings are also limited. The paper notes that this limitation didn't seem to hinder its performance in this task.
  • Future Work: The authors make their code publicly available, encouraging further research into improving dialog act recognition, particularly in scenarios with limited or no punctuation. They also suggest investigating how automatically retrieved punctuation might help mitigate performance drops.

Summary updated

Abstract

Dialog acts can be interpreted as the atomic units of a conversation, more fine-grained than utterances, characterized by a specific communicative function. The ability to structure a conversational transcript as a sequence of dialog acts—dialog act recognition, including the segmentation—is critical for understanding dialog. We apply two pre-trained transformer models, XLNet and Longformer, to this task in English and achieve strong results on Switchboard Dialog Act and Meeting Recorder Dialog Act corpora with dialog act segmentation error rates (DSER) of 8.4% and 14.2%. To understand the key factors affecting dialog act recognition, we perform a comparative analysis of models trained under different conditions. We find that the inclusion of a broader conversational context helps disambiguate many dialog act classes, especially those infrequent in the training data. The presence of punctuation in the transcripts has a massive effect on the models’ performance, and a detailed analysis reveals specific segmentation patterns observed in its absence. Finally, we find that the label set specificity does not affect dialog act segmentation performance. These findings have significant practical implications for spoken language understanding applications that depend heavily on a good-quality segmentation being available.
BibTeX
@article{zelasko2021what,
  author = {Piotr Zelasko and Raghavendra Pappagari and Najim Dehak},
  title = {{What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Dqh9fF4rXcu2},
}
Figure