Dual Process Masking for Dialogue Act Recognition

Yeo Jin Kim, Halim Acosta, Wookhee Min, Jonathan P. Rowe, Bradford W. Mott, Snigdha Chaturvedi, James C. LesterEMNLP 2024 · DOI

Abstract

Dialogue Act Recognition (DAR) is essential for understanding conversational intent but faces significant challenges in data-scarce domains and model interpretability. We present Dual-Process Masking (DP-Masking), a framework inspired by cognitive psychology that iteratively masks irrelevant tokens to focus on the functional "skeleton" of utterances. By simulating reflective (System 2) and intuitive (System 1) learning, DP-Masking significantly improves performance in low-resource settings and provides insights into the linguistic structures driving model decisions.

Introduction

Dialogue Act Recognition involves labeling the communicative function of a sentence—such as asking a question, sharing an idea, or negotiating. While Large Language Models (LLMs) excel with abundant data, they often struggle in specialized fields like education or healthcare where labeled data is scarce. Furthermore, these models are frequently "black boxes," making it difficult to discern which specific words trigger a label.

Existing masking techniques, such as those used in BERT, are typically random or fixed. DP-Masking addresses these issues by introducing a task-specific, interpretable masking strategy that "denoises" input by de-emphasizing tokens that do not contribute to communicative intent.

Comparison of standard classification versus DP-Masking, illustrating how removing conversational noise leads to correct label prediction.

Methodology: The Dual-Process Framework

The DP-Masking framework draws inspiration from Dual-Process Theory, which posits two systems of human thought: a fast, intuitive System 1 and a slow, deliberate System 2.

The framework implements an iterative loop consisting of two primary stages:

  1. The Reflective Stage (System 2): This stage involves deliberate experimentation. A Masker model generates various token masks, and a Classifier evaluates whether these masks maintain or improve labeling accuracy. Successful masks—those that hide irrelevant words without degrading performance—are stored in a Working Memory.
  2. The Intuitive Stage (System 1): This stage internalizes the refined knowledge. A final classifier is trained using the optimized masking patterns retrieved from the Working Memory.
The DP-Masking architecture, showing the iterative loop between the Masker, Working Memory, and Classifier across reflective and intuitive learning stages.

By the end of training, the model learns to "see through" conversational filler, focusing on the structural elements that define the dialogue act.

Experimental Setup and Datasets

The framework was evaluated across four diverse datasets to test its robustness:

  • CPS (Collaborative Problem-Solving): A small, specialized dataset (~2k instances) of students solving science mysteries, used to test performance in data-scarce environments.
  • Oasis: 1,219 transcripts of help-desk interactions.
  • TalkMoves: A large-scale dataset (170k instances) of K-12 classroom discourse.
  • SwDA (Switchboard Dialog Act): A benchmark of 199k telephone conversation instances.

Key Results and Insights

DP-Masking achieved state-of-the-art (SOTA) results on the CPS and Oasis datasets. On the small CPS dataset, the framework outperformed "no-masking" baselines by an average of 17.6% in macro F1 score.

Data Efficiency and Model Scaling

The benefits of DP-Masking were most pronounced in low-resource scenarios (e.g., at the 5% data mark). While the performance gap narrowed as dataset sizes approached 100%, the framework proved to be a powerful tool for scenarios where labeled data is expensive. Interestingly, smaller models like Flan-T5 (80M parameters) benefited more than massive models like Llama-3, likely due to their ability to undergo more comprehensive fine-tuning on limited hardware.

Strategic Silence and Interpretability

The model demonstrated "strategic silence" by learning which words to ignore. While it frequently masked common "stop words" like "I'm," it rarely masked prepositions like "at" or "in," identifying them as critical structural signals for specific dialogue acts. This behavior provides a clearer window into the "functional language structure" the model uses to distinguish between intents, such as "negotiating" versus "regulating."

Mastery of Short Utterances

DP-Masking showed a significant 17.4% accuracy improvement on short utterances. In brief sentences where every word is impactful, removing even a single piece of "noise" prevents the model from being misled by frequent but irrelevant tokens.

Limitations and Future Work

The primary drawback of DP-Masking is its computational cost; training four components across multiple iterations makes it approximately seven times slower than standard fine-tuning. Additionally, the current implementation masks tokens independently. Future research may explore "collocation-aware" masking and extending the dual-process approach to other NLP tasks such as summarization or question answering.

Summary updated

BibTeX
@inproceedings{kim2024dual,
  author = {Yeo Jin Kim and Halim Acosta and Wookhee Min and Jonathan P. Rowe and Bradford W. Mott and Snigdha Chaturvedi and James C. Lester},
  title = {{Dual Process Masking for Dialogue Act Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.895},
}
Figure