Portability of semantic annotations for fast development of dialogue corpora

Bassam Jabaian, Fabrice Lefèvre, Laurent BesacierInterspeech 2012

TL;DR

This paper tackles the challenge of rapidly developing spoken dialogue systems for new languages and domains. They propose a semi-automatic annotation process that leverages existing corpora to pre-annotate new data, significantly reducing manual effort. This approach leads to substantial productivity gains and enables faster creation of valuable datasets for training spoken language understanding (SLU) models.

What Problem Does This Paper Address?

Developing robust spoken dialogue systems (like virtual assistants or customer service bots) requires large amounts of annotated data. This data is crucial for training the Spoken Language Understanding (SLU) component, which extracts meaning from transcribed user utterances. However, collecting and annotating this data is a time-consuming and expensive bottleneck, especially when adapting systems to new languages or entirely new application domains. Existing methods often rely on having a fully deployed system to iteratively collect data, which isn't always feasible. This paper addresses the need for a faster, more efficient way to create annotated dialogue corpora for new languages and domains, starting from a fixed dataset.

Methods

The core of their approach is a semi-automatic annotation process that involves two main steps:

  1. Automatic Pre-annotation: They leverage an existing SLU system trained on a well-annotated corpus (the French MEDIA corpus) to automatically tag new, untranscribed dialogue data.
  2. For new languages (PM-LANG): They translate the existing French MEDIA corpus into the target language (Italian in this case) using statistical machine translation (SMT). Then, they transfer the semantic annotations from the French to the translated Italian corpus. This transferred annotation is used to train an SLU tagger for the new language.
  3. For new domains (PM-DOM): They adapt the existing French MEDIA SLU tagger to the new domain (theatre ticket reservation) by incorporating a named entity tagger that recognizes domain-specific terms (like play titles and artist names).
  4. Manual Correction: Human annotators then review and correct the automatically generated annotations.

The authors use Conditional Random Fields (CRFs), a statistical method well-suited for sequential tagging tasks, to build their SLU taggers. They also introduce a protocol for measuring productivity gains by comparing the time taken to correct pre-annotations versus annotating from scratch.

Datasets

  • French MEDIA Corpus: The primary source corpus, containing hotel reservation and tourist information dialogues. It's used as the foundation for pre-annotation.
  • PM-LANG Corpus: A newly created corpus for Italian, based on the MEDIA domain (hotel reservations).
  • PM-DOM Corpus: A newly created corpus for French, focused on the theatre ticket reservation domain.

The paper describes the creation of these new corpora and their division into training and testing sets for evaluation.

Key Insights and Ideas

  • Portability is Key: The paper demonstrates that semantic annotations can be "ported" from one language and domain to another, albeit with varying degrees of success and requiring different adaptation strategies.
  • Pre-annotation is a Game-Changer: Automatically pre-annotating data significantly reduces the manual effort required for annotation, leading to substantial productivity gains. This is a crucial finding for accelerating corpus development.
  • Translation for Language Portability: Translating an existing corpus and transferring annotations is an effective strategy for bootstrapping SLU systems in new languages.
  • Domain Adaptation Requires More: For new domains, simply using an out-of-domain tagger isn't enough; incorporating domain-specific knowledge (like named entities) is essential for effective pre-annotation.
  • Combining Corpora Can Help (Sometimes): Merging the original and newly created corpora can further improve SLU model performance, especially for new languages, but requires careful handling to avoid negative interference in new domains.

Key Results

  • Productivity Gains: The semi-automatic annotation process achieved significant productivity gains, with over 50% for PM-LANG and 40% for PM-DOM over three iterations of refinement.
  • SLU Performance:
    • For the Italian (PM-LANG) corpus, the developed SLU model achieved a Concept Error Rate (CER) of 18.9% on its test set.
  • For the theatre ticket reservation (PM-DOM) corpus, the SLU model achieved a CER of 19.1%.
  • Combination Benefits: Combining the new corpora with the original MEDIA corpus led to improved performance, particularly for the Italian task.

Limitations and Future Work

  • Domain Interference: Combining the MEDIA corpus with the PM-DOM corpus without filtering led to a degradation in performance. This highlights the challenge of combining data from significantly different domains.
  • Filtering Strategies: The paper explores filtering the MEDIA corpus to improve combination with PM-DOM, finding that a more aggressive filtering (F'-MEDIA) yielded better results, suggesting that removing noisy data is beneficial.
  • Future Work: The authors suggest further investigation into more sophisticated methods for combining corpora and improving the robustness of the pre-annotation process. They also implicitly point towards the need for better strategies to handle concept mismatches between domains.

Summary updated

BibTeX
@inproceedings{jabaian2012portability,
  author = {Bassam Jabaian and Fabrice Lefèvre and Laurent Besacier},
  title = {{Portability of semantic annotations for fast development of dialogue corpora}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure