The SAMMIE Multimodal Dialogue Corpus Meets the Nite XML Toolkit

Ivana Kruijff-Korbayová, Verena Rieser, Ciprian Gerstenberger, Jan Schehl, Tilman BeckerNLPXML@EACL 2006 · OpenReview

Abstract

Ivana Kruijff-Korbayová, Verena Rieser, Ciprian Gerstenberger, Jan Schehl, Tilman Becker. Proceedings of the 5th Workshop on NLP and XML (NLPXML-2006): Multi-Dimensional Markup in Natural Language Processing.

2006.

Introduction

Developing natural multimodal dialogue systems—where users interact via speech and touchscreens while multitasking—presents significant data representation challenges. The SAMMIE project addresses the difficulty of capturing overlapping information layers, such as spoken words, syntactic structures, and visual feedback, which often do not align in a simple hierarchy. Traditional XML formats struggle with these "overlapping spans," necessitating a more robust framework for storing and cross-referencing disparate data streams while maintaining temporal relationships.

Methodology and Data Representation

The researchers utilized a Wizard-of-Oz (WOZ) experimental design to observe natural human behavior. To manage the complex data generated, they adopted the Nite XML Toolkit (NXT). This framework supports stand-off annotation, where annotations are stored in separate XML files pointing to base data, allowing independent feature annotation (e.g., prosody vs. dialogue acts).

Crucially, NXT enables multi-rooted tree structures, allowing for multiple hierarchies. This is essential for multimodal data where a dialogue "turn" might contain speech "words" but also link to non-speech "graphical events."

NXT-based visualization of overlapping annotation layers

The SAMMIE Corpus

The SAMMIE corpus was collected in two phases:

  • SAMMIE-1: Involved 24 subjects in speech-only interactions.
  • SAMMIE-2: Involved 35 subjects in a multimodal setup (speech + screen), including a "driving simulator" condition to evaluate cognitive load.

The sessions involved tasks like music searching and playlist management in German, using an English music database. In SAMMIE-2, wizards were given the autonomy to choose between speech, screen output, or both, capturing the strategic nuances of multimodal interaction.

Annotation and Results

A key focus was the reliable annotation of Clarification Requests (CRs). The team developed specialized Java-based GUI tools to help annotators classify these complex phenomena. They achieved a high inter-annotator agreement (Kappa of 0.788) for CR identification.

Custom annotation tool for Clarification Requests (CRs)

The researchers also utilized visualization tools like Amigram to inspect dependencies across layers, such as aligning specific screen events with spoken words. The resulting corpus features a rich set of layers including words, graphical output, utterances, turns, domain objects, and dialogue acts.

Conclusion and Future Work

While some manual annotations like referring expressions were still being finalized at the time of publication, the SAMMIE corpus provides a robust infrastructure for multimodal research. Future work aims to leverage this dataset to train reinforcement learning models for autonomous dialogue strategy selection. Although currently specific to German, the framework demonstrates a scalable approach to multi-dimensional markup in NLP.

Summary updated

BibTeX
@inproceedings{kruijffkorbayova2006sammie,
  author = {Ivana Kruijff-Korbayová and Verena Rieser and Ciprian Gerstenberger and Jan Schehl and Tilman Becker},
  title = {{The SAMMIE Multimodal Dialogue Corpus Meets the Nite XML Toolkit}},
  booktitle = {NLPXML@EACL},
  year = {2006},
  url = {https://openreview.net/forum?id=BTDWPx6Rpz5},
}
Figure