TL;DR
This paper introduces a new, richly annotated human-human dialogue corpus for train timetable inquiries. By combining and extending existing dialogue act tagging schemes, the authors provide a valuable resource for training more robust spoken dialogue systems, particularly for mixed-initiative interactions.
What Problem Does This Paper Address?
Developing effective spoken dialogue systems (SDSs) requires large, high-quality datasets. While many systems are trained on human-computer (HC) dialogues, these often suffer from a fixed dialogue policy, limiting the system's ability to learn robust conversational strategies. Human-human (HH) dialogues, on the other hand, offer more natural and flexible interactions, but are more complex to annotate. Specifically, the authors aim to address the need for a corpus that can support both speech recognition (ASR) and dialogue management (DM) components of SDSs, especially for mixed-initiative dialogues where the user has more control over the conversation flow. They also highlight the challenge of creating a semantic annotation scheme that is both comprehensive and easy for annotators to use consistently.
Methods
The core contribution lies in the creation and annotation of a new dialogue corpus. The authors built upon existing dialogue act tagging schemes, specifically DATE (Dialogue Act Tagging for Evaluation) and DAMSL (Dialogue Act Markup in Several Layers), to create a hybrid approach. They adopted DATE's three-dimensional structure (DOMAIN, SPEECH-ACT, SEMANTICS) but replaced the TASK-SUBTASK dimension with a more detailed SEMANTICS dimension. They also incorporated specific tags from DAMSL (like agreement, politeness, and speech-repair) to better capture the nuances of HH dialogues.
The annotation process involves:
- Orthographic Transcription: Recording and transcribing the spoken dialogues.
- Normalization: Creating normalized transcriptions, including named entities.
- Dialogue Act Tagging: Assigning dialogue acts to utterances based on the developed hybrid tagset.
- Abstract Semantic Annotation: Annotating utterances with task-relevant semantic information (e.g., departure time, destination).
A custom annotation tool was developed to facilitate this process, offering features for configurable tagsets, annotation validation, and dialogue overview.
Datasets
The paper introduces a newly developed human-human train timetable dialogue corpus.
- Collection Period: April 2000 to September 2000.
- Size: 6584 recorded calls, with 6353 transcribed dialogues.
- Content: 106 hours of speech, single channel, 8kHz sampling, A-Law compression.
- Characteristics: Average dialogue length of 60 seconds. User speech is 32%, operator speech is 39%. Contains 81,543 turns. Vocabulary size is approximately 12k words.
- Annotation: Includes orthographic transcription, normalized transcription, normalized transcription with named entities, and dialogue act tags with abstract semantic annotation.
This corpus is a significant extension of previous work on a train timetable information center dialogue corpus.
Key Insights and Ideas
- Hybrid Tagging Scheme: Combining DATE and DAMSL offers a robust and flexible approach to dialogue act annotation, capturing both task-specific information and conversational nuances essential for HH dialogues.
- SEMANTICS Dimension: Replacing DATE's TASK-SUBTASK with a dedicated SEMANTICS dimension allows for richer, abstract semantic annotation, crucial for training semantic decoders.
- Mixed-Initiative Dialogue Focus: The HH nature of the corpus naturally lends itself to studying mixed-initiative dialogues, which are more complex for dialogue management than system-initiative dialogues.
- Annotation Simplicity: The abstract semantic annotation is designed to be simple and unambiguous, reducing the need for extensive linguistic expertise from annotators and improving annotation consistency.
- Practical Applicability: The corpus is intended to be directly usable for training statistical models for both SLU and DM components of SDSs.
Key Results
The paper describes the development and annotation process of the corpus, rather than presenting quantitative results from trained models. The key outcome is the creation of a valuable, richly annotated dataset. The authors report that they have annotated 84 dialogues by their dialogue act tagging scheme as of January 2005. They anticipate that this corpus will be used to train a semantic decoder using Hidden Vector State (HVS) parsers and a dialogue act decoder using dynamic Bayes networks.
Limitations and Future Work
- Limitations:
- The authors acknowledge that while they aimed for robustness, they have not yet formally evaluated inter-annotator agreement for their new scheme.
- Some callers were from abroad with heavily accented Czech, making their recordings unsuitable for further processing.
- Future Work:
- Evaluate inter-annotator agreement and compare it with existing DATE or DAMSL-based schemes.
- Develop a dialogue act decoder using dynamic Bayes networks.
- Compose both dialogue act and semantic decoders into a finite state transducer.
- Build an automatic dialogue evaluator based on the PARADISE framework.
- Use the corpus for training a mixed-initiative spoken dialogue system.
Summary updated