Mokusei: a telephone-based Japanese conversational system in the weather domain

Mikio Nakano, Yasuhiro Minami, Stephanie Seneff, Timothy J. Hazen, D. Scott Cyphers, James R. Glass, Joseph Polifroni, Victor ZueInterspeech 2001 · DOI

TL;DR

This paper introduces MOKUSEI, a Japanese conversational system for accessing weather information over the phone. It builds upon the English JUPITER system, demonstrating a successful multilingual approach by adapting existing components and introducing novel solutions for Japanese language processing, particularly in natural language understanding and generation.

What Problem Does This Paper Address?

Developing multilingual conversational systems presents a significant challenge. While many systems are built for English, extending them to other languages requires addressing fundamental linguistic differences. The authors tackle the problem of creating a Japanese counterpart to their established English weather information system, JUPITER. This involves adapting speech recognition, natural language understanding (NLU), and natural language generation (NLG) components to the nuances of the Japanese language, which has different grammatical structures and phonological characteristics compared to English. A key challenge is ensuring that the system can effectively process Japanese utterances and generate natural-sounding responses, especially given the lack of readily available Japanese language resources and established standards for certain linguistic tasks.

Methods

MOKUSEI largely reuses the architecture and many components of the English JUPITER system, leveraging the Galaxy Communicator architecture for inter-component communication.

  • Speech Recognition: The SUMMIT speech recognition system was adapted. While the core system required no design adjustments, significant effort went into creating a Japanese vocabulary (1,151 words), phonetic pronunciations, and a class n-gram language model with 56 word classes. Phonological rules were developed to handle variations common in Japanese, and acoustic models were initially trained on English data, then retrained with a combination of English and Japanese data as it became available.
  • Natural Language Understanding (NLU): The TINA system, originally developed for English, was adapted for Japanese. A key innovation was the use of a "trace" mechanism to efficiently handle Japanese's left-recursive grammatical structures. This mechanism allows for shallow parsing initially, with constituents being moved later to their appropriate positions after their role is identified by subsequent particles, effectively reordering constructs to be right-recursive and significantly reducing computational requirements.
  • Natural Language Generation (NLG): The GENESIS-II generation system was redesigned to improve the quality of Japanese translations of weather reports. A novel "pull" mechanism was introduced, allowing constituents to pre-generate other constituents among their descendants, enabling more flexible and context-aware sentence restructuring than literal translation. This was crucial for generating natural Japanese sentences where word order and phrasing differ significantly from English.
  • Dialogue Management and Synthesis: The dialogue manager was largely identical to JUPITER's, with the primary adaptation being the default to the Celsius scale for temperatures. Japanese speech synthesis was handled by NTT's FLUET software, integrated via a wrapper.
  • Content Processing: The weather database was expanded to include 144 Japanese cities, with data sourced from web sites and a commercial distribution service.

Datasets

  • Training Data: Initial acoustic models were trained on English utterances. As Japanese data became available, acoustic models were retrained using a combination of English and Japanese data. The language model was trained on naive user utterances.
  • Evaluation Data: The paper reports on data collected from naive users. A training set comprised 542 dialogues and 8,038 utterances, while a test set contained 168 dialogues and 2,442 utterances. Additional read speech utterances (1,900) and expert user utterances (2,592) were used to augment the training of acoustic models.

Key Insights and Ideas

  • Multilingualism via Language-Independent Semantics: The core idea that a common, language-independent semantic representation can be extracted from input utterances is validated for a restricted domain like weather information. This approach significantly simplifies the development of multilingual systems.
  • Adapting English NLU for Japanese Recursion: The successful adaptation of TINA's "trace" mechanism to handle Japanese's left-recursive grammar is a significant technical achievement. This demonstrates that established parsing techniques can be modified to accommodate diverse linguistic structures without a complete overhaul.
  • "Pull" Mechanism for Natural Generation: The redesign of the GENESIS-II generation system with a "pull" mechanism is crucial for generating fluent and natural-sounding Japanese. This highlights the importance of flexible generation strategies that go beyond literal translation to capture idiomatic expressions and sentence structures.
  • Data-Driven Improvement: The paper emphasizes the iterative process of data collection and system improvement. Initial performance was limited by the lack of training data, but subsequent data collection and retraining led to significant improvements.

Key Results

  • Speech Recognition: The current recognizer has an active vocabulary of 1,151 words with a trigram test set perplexity of 13.0. The word error rate on in-vocabulary test data (1,745 utterances) was 8.5%, with a sentence error rate of 33.1%. On the complete test set, these rates increased to 19.0% and 45.9%, respectively.
  • Natural Language Understanding: The concept error rate for parsable test set utterances (1,515) was 12.0%. The system covers over 75% of naive user utterances that do not include artifacts.
  • Natural Language Generation: Over 90% of manually transcribed test set utterances could be considered fluent.
  • User Data: Over 700 naive users contributed to data collection, resulting in approximately 10,480 utterances from 713 calls.

Limitations and Future Work

  • Limited Vocabulary: While the vocabulary of 1,151 words is relevant to the weather domain, it is still a limitation. Future work will likely involve expanding this vocabulary.
  • Performance on Unseen Data: The performance metrics show a significant increase in error rates on the complete test set compared to the in-vocabulary set, indicating that the system still struggles with out-of-vocabulary words or less common phrasing.
  • Ongoing Data Collection: The authors explicitly state that ongoing data collection is crucial for further performance improvement. They believe that continued data collection at both MIT and NTT will lead to even better results.
  • Multilingual Framework Validation: The paper positions MOKUSEI, alongside the Mandarin Chinese version (MUXING), as evidence that the JUPITER architecture is a viable framework for multilingual conversational system research, suggesting future work in extending this framework to other languages and domains.

Summary updated

BibTeX
@inproceedings{nakano2001mokusei,
  author = {Mikio Nakano and Yasuhiro Minami and Stephanie Seneff and Timothy J. Hazen and D. Scott Cyphers and James R. Glass and Joseph Polifroni and Victor Zue},
  title = {{Mokusei: a telephone-based Japanese conversational system in the weather domain}},
  booktitle = {Interspeech},
  year = {2001},
  doi = {10.21437/eurospeech.2001-344},
}
Figure