MiPad: a multimodal interaction prototype

Xuedong Huang, Alejandro Acero, Ciprian Chelba, Li Deng, Jasha Droppo, D. Duchene, J.t. Goodman, Hsiao-Wuen Hon, Derek Jacoby, Lì Jiāng, R. Loynd, M. Mahajan, Peter Mau, S. Meredith, S. Mughal, S. Neto, Mike Plumpe, Keith Steury, Gina Venolia, Kaidong Wang, Y. Wang

Abstract

Dr. Who is a Microsoft research project aiming at creating a speech-centric multimodal interaction framework, which serves as the foundation for the NET natural user interface. MiPad is the application prototype that demonstrates compelling user advantages for wireless personal digital assistant (PDA) devices, MiPad fully integrates continuous speech recognition (CSR) and spoken language understanding (SLU) to enable users to accomplish many common tasks using a multimodal interface and wireless technologies. It tries to solve the problem of pecking with tiny styluses or typing on minuscule keyboards in today's PDAs. Unlike a cellular phone, MiPad avoids speech-only interaction. It incorporates a built-in microphone that activates whenever a field is selected. As a user taps the screen or uses a built in roller to navigate, the tapping action narrows the number of possible instructions for spoken word understanding. MiPad currently runs on a Windows CE Pocket PC with a Windows 2000 machine where speech recognition is performed. The Dr Who CSR engine uses a unified CFG and n-gram language model. The Dr Who SLU engine is based on a robust chart parser and a plan-based dialog manager. The paper discusses MiPad's design, implementation work in progress, and preliminary user study in comparison to the existing pen-based PDA interface.
BibTeX
@inproceedings{huang2001mipad,
  author = {Xuedong Huang and Alejandro Acero and Ciprian Chelba and Li Deng and Jasha Droppo and D. Duchene and J.T. Goodman and Hsiao-Wuen Hon and Derek Jacoby and Lì Jiāng and R. Loynd and M. Mahajan and Peter Mau and S. Meredith and S. Mughal and S. Neto and Mike Plumpe and Keith Steury and Gina Venolia and Kaidong Wang and Y. Wang},
  title = {{MiPad: a multimodal interaction prototype}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2001},
  doi = {10.1109/icassp.2001.940754},
}

Loading related content...

Figure