Dialogue Act Tagging for Instant Messaging Chat Sessions

Edward IvanovicACL 2005

Abstract

Instant Messaging (IM) presents unique challenges for Dialogue Act (DA) tagging due to its quasi-synchronous nature, where messages can be out of order or contain multiple utterances. This paper proposes a statistical approach combining Naive Bayes classifiers with n-gram discourse models, supplemented by a novel "typing rate" algorithm to handle message synchronization. Experimental results on a dataset of online shopping assistance dialogues show that a bigram model achieves 81.6% accuracy, significantly outperforming baseline methods.

Introduction: The Challenge of IM-Specific NLP

Dialogue act tagging—the process of identifying the functional intent behind an utterance (e.g., a question, statement, or request)—is a foundational task in natural language processing. While well-studied in the context of spoken speech, Instant Messaging (IM) introduces distinct complexities. Unlike linear spoken conversations, IM is "quasi-synchronous."

Two primary hurdles define IM dialogue:

  1. Segmentation: A single message may contain multiple distinct dialogue acts, or conversely, one dialogue act may span multiple messages.
  2. Synchronization: Typing delays often result in "out-of-order" flows. A user might respond to a message from several turns ago while their interlocutor is simultaneously typing a new one.
Sample IM dialogue illustrating turn-taking, message timing, and dialogue act tagging.

Methodology

The proposed system utilizes a hybrid statistical model to classify intents:

  • Naive Bayes Classifier: Acts as a "likelihood" model, analyzing the "bag-of-words" within an utterance. For instance, phrases like "how do I" strongly correlate with OPEN-QUESTION tags.
  • N-gram Discourse Model: Captures the sequential probability of dialogue acts. For example, a YES-NO-QUESTION is statistically likely to be followed by a YES-ANSWER or NO-ANSWER.

Addressing Synchronization: The Typing Rate Algorithm

To mitigate the synchronization problem, the author introduced a "typing rate" heuristic. By calculating the time elapsed between messages relative to message length, the system determines if a message is a direct response to its immediate predecessor or an earlier utterance. If the calculated typing rate is physically impossible for a human, the system assumes the user began typing before the previous message was received, allowing the conversation to be re-threaded correctly before DA tagging.

Dataset and Annotation

The study utilized a custom corpus of online shopping assistance dialogues from MSN Shopping. The dataset features interactions between real customers and shopping assistants. Three linguistics graduate students manually labeled the data using a refined set of 12 dialogue act tags (adapted from the 42-tag DAMSL set). The annotation process achieved a Kappa statistic of 0.87, indicating high inter-annotator reliability.

Key Results and Analysis

The models were evaluated using 9-fold cross-validation. The results demonstrate that lexical content is a powerful indicator of intent in task-oriented IM, with the Naive Bayes model alone reaching 80.1% accuracy.

Performance metrics for various dialogue act tagging models compared against a baseline.

The bigram model, which incorporates context from the previous dialogue act, emerged as the top performer with 81.6% accuracy. Interestingly, the trigram model showed a performance dip (80.9%), likely due to data sparsity issues where the dataset lacked sufficient three-act sequences for robust training. Furthermore, the "typing rate" algorithm achieved 94.2% accuracy in identifying message dependencies, a significant improvement over the 80% baseline of simple linear assumption.

Conclusion and Future Directions

This research confirms that while IM lacks the prosodic cues of speech, temporal data like typing rates can serve as effective proxies for conversation flow. Future work should focus on automating the segmentation of utterances within messages and exploring more complex models like Dialogue Macrogame Theory to handle nested interactions. Additionally, larger annotated corpora will be essential to overcome the limitations of data sparsity in higher-order n-gram models.

Summary updated

BibTeX
@inproceedings{ivanovic2005dialogue,
  author = {Edward Ivanovic},
  title = {{Dialogue Act Tagging for Instant Messaging Chat Sessions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2005},
}
Figure