Abstract
Neural network-based chatbots often operate as "black boxes," complicating efforts to understand or control their outputs. This research proposes a method to predict the Dialogue Act (DA)—the functional intent of an utterance—of a future response by analyzing conversation history. By predicting intent prior to text generation, the researchers aim to enhance the interpretability and controllability of AI dialogue systems. Utilizing a multi-encoder architecture on the Switchboard Dialogue Act (SwDA) corpus, the proposed model achieves a 69.7% accuracy, significantly outperforming baselines that rely solely on immediate context.
Introduction
Most contemporary chatbots utilize sequence-to-sequence (Seq2Seq) models that map inputs directly to outputs. While effective for fluency, these models offer limited insight into the underlying rationale for a specific response. This lack of transparency poses challenges for applications requiring high safety and brand consistency.
To address this, the authors propose predicting the Dialogue Act (DA) of the upcoming, unseen response. Unlike previous work that focuses on identifying the DA of existing utterances, this task requires the model to anticipate future intent. This is inherently difficult because brief utterances, such as backchannels ("Uh-huh"), provide minimal information about the subsequent turn without broader conversational context.
Methodology
The proposed architecture employs three distinct encoders to capture various dimensions of a dialogue:
- Utterance Encoder: A Recurrent Neural Network (RNN) using Gated Recurrent Units (GRUs) to transform individual utterances into vector representations.
- Context Encoder: An RNN that processes the sequence of utterance vectors to track the conversation's history. It incorporates a "speaker change tag" to signal shifts in the conversational floor.
- Dialogue-act (DA) Encoder: This encoder processes the sequence of preceding DAs (e.g., Statement → Question). Given the limited vocabulary of DAs compared to natural language, this component is robust against data sparseness.
The outputs from the Context and DA encoders are concatenated and passed through a final classifier (a fully-connected layer with softmax) to predict the next DA. The model was trained using teacher forcing, providing the "gold standard" previous DA during training to prevent error propagation.
Experimental Results
The model was evaluated using the Switchboard Dialogue Act (SwDA) corpus. The original complex tag set was consolidated into nine broad categories: Statement, Uninterpretable, Understanding, Agreement, Directive, Greeting, Question, Apology, and Other. The dataset comprised over 1 million utterances.
The results demonstrate that incorporating conversation history and DA sequences is vital for accurate prediction. The proposed model significantly outperformed baselines that considered only the most recent utterance or lacked specific encoders.
Key performance metrics include:
- Accuracy: 69.7%, a 3.0% improvement over the utterance-only baseline.
- Macro-F1 Score: 32.4%, representing a substantial 10.8% increase over the baseline.
- Infrequent Tag Performance: The model showed dramatic improvements in predicting rare DAs. For instance, the F1-score for "Greetings" rose from 46.7 (text-only) to 81.3.
Discussion and Conclusion
The study highlights that the immediate previous utterance is often insufficient for predicting future intent; broader history is mandatory. The DA sequence acts as a structural "cheat sheet," providing a cleaner map of the conversation than raw text alone.
However, limitations remain. The model struggles with long-range semantic dependencies that require deep world knowledge—exemplified by failure cases in technical discussions (e.g., the "Aerosol" problem). Future work will focus on integrating these predicted DAs into text generation models to produce more coherent and controllable dialogue, as well as accounting for individual speaker styles.
Summary updated