Investigating multiple approaches for SLU portability to a new language

Bassam Jabaian, Laurent Besacier, Fabrice LefèvreInterspeech 2010

TL;DR

This paper tackles the challenge of making spoken language understanding (SLU) systems work in new languages without starting from scratch. They propose and evaluate three methods for "porting" an SLU system from French to Italian, leveraging statistical machine translation (SMT) and word alignment techniques. The findings suggest that porting at the test level, by translating user queries, offers the most promising results with minimal performance degradation.

What Problem Does This Paper Address?

Building robust Spoken Language Understanding (SLU) systems typically requires large amounts of annotated data specific to a domain and language. This annotation process is time-consuming and expensive. The core problem this paper addresses is how to efficiently adapt an existing SLU system, developed for one language (the source language, French in this case), to function in a different language (the target language, Italian) with minimal human effort and cost. This is crucial for scaling SLU technology across diverse linguistic landscapes.

Methods

The authors investigate three distinct approaches for SLU portability, all aiming for automatic translation and avoiding manual post-editing of translated data:

  • Test Translation: This is the simplest approach. An existing SLU system trained on the source language (French) is kept as is. When a query comes in the target language (Italian), it's automatically translated into the source language using an SMT system, and then fed into the original French SLU system. The performance hinges on the quality of the target-to-source SMT system.
  • Tagged Translation: This method involves building a new SLU system for the target language. The source language training corpus, already annotated with semantic concepts, is translated into the target language. Crucially, XML tags are used during the SMT process to preserve the original chunk boundaries and semantic annotations. This allows for the creation of an annotated target language corpus, which is then used to train a new SLU model in the target language.
  • Alignment: This approach bypasses SMT for the training data translation. Instead, it relies on word-to-word alignments between the source and target language parallel corpus. An algorithm uses these alignments and the source corpus's chunk boundaries to infer semantic concepts and their corresponding words in the target language, effectively creating an annotated target corpus for training. The authors also explore an extension of this method by enriching the target language words with their aligned source language words to act as a form of "POS tagging" for disambiguation.

Datasets

  • MEDIA Corpus (French): This is the primary dataset used. It's a French dialogue corpus covering hotel reservation and tourism information. It's divided into training (approx. 13k sentences), development (approx. 1.3k sentences), and evaluation (approx. 3.5k sentences) sets.
  • Italian Manual Translation: A portion of the French MEDIA training set (approx. 5.6k sentences), along with the development and test sets, was manually translated into Italian for the LUNA project. This manual translation serves as a parallel corpus for training SMT systems and as a gold standard for evaluation.
  • Italian Automatic Translation: The remaining parts of the French MEDIA training corpus were automatically translated into Italian using the developed SMT systems.

Key Insights and Ideas

  • Portability is Achievable: The study demonstrates that porting SLU systems to a new language is feasible with automatic methods, achieving performance only a few percentage points below the source language system.
  • Test Translation is Surprisingly Effective: The simplest approach, translating the input query at test time, yields the best results. This suggests that the core SLU logic is robust enough to handle translated inputs, and the SMT system's quality is the primary bottleneck.
  • Preserving Annotations is Key: For training-level porting, maintaining semantic annotations during translation is critical. The "Tagged Translation" method, using XML tags to enforce chunk boundaries, shows promise.
  • Word Alignments Offer a Data-Driven Path: The "Alignment" method, leveraging word-to-word alignments, provides a way to infer target language annotations without relying on manual translation of the entire training corpus, offering a more scalable solution.
  • Enriching Target Data Helps: Adding information from the source language (like aligned words) to the target language training data can improve SLU performance, especially when training data is limited. This acts as a form of noisy disambiguation.

Key Results

  • Test Translation: Achieved the lowest Concept Error Rate (CER) for Italian SLU systems, with CERs of 19.9% (on a partial corpus) and 12.9% (on the full corpus), closely approaching the French baseline (18.5% and 12.1% respectively).
  • Tagged Translation: Performed slightly worse than Test Translation, with CERs of 23.1% (partial) and 22.7% (full).
  • Alignment: Showed competitive results, with CERs of 22.8% (partial) and 20.5% (full).
  • Alignment with Source Word Enrichment: Improved the alignment method's performance, reducing CER to 21.2% (partial) and 20.1% (full).
  • SMT Performance: The French-to-Italian SMT system achieved a BLEU score of 43.62, and Italian-to-French achieved 47.18, measured against manually translated test sets.

Limitations and Future Work

  • Clean Test Set: The authors acknowledge that the evaluation was performed on a clean test set (manually transcribed). They plan to reproduce these results on noisy data from speech recognizer outputs, which is a more realistic scenario.
  • SMT System Improvement: The performance of the portability methods is directly tied to the quality of the SMT systems. Further improvements in SMT could lead to even better SLU portability.
  • Manual Revision: While the paper focuses on fully automatic methods, the authors note that manual post-editing of translations, as done in prior work, could potentially yield better results but at a higher cost.
  • Ambiguous Alignments: The "Alignment" method faces challenges with ambiguous word alignments, where a target word aligns to multiple source words belonging to different concepts. The proposed strategy of using the first aligned source word is a heuristic that might not always be optimal.

Summary updated

BibTeX
@inproceedings{jabaian2010investigating,
  author = {Bassam Jabaian and Laurent Besacier and Fabrice Lefèvre},
  title = {{Investigating multiple approaches for SLU portability to a new language}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure