TL;DR
This paper presents a novel approach to building a Japanese Input Method Editor (IME) for mobile devices by leveraging massive web data. They introduce an online algorithm for mining new compound words and a multi-stage filtering process to create a compact yet highly accurate lexicon, outperforming existing state-of-the-art systems.
What Problem Does This Paper Address?
Typing in languages like Japanese on Latin keyboards is challenging due to the vast number of characters and word combinations. Traditional Input Method Editors (IMEs) require extensive lexicons to map phonetic inputs (Kana) to correct character sequences (Kanji). However, mobile devices have limited memory and processing power, making it difficult to store large lexicons and perform complex computations. This paper addresses the challenge of creating a highly effective Japanese IME for mobile devices that can handle the complexity of the language while respecting device constraints, by efficiently learning from the vast and ever-evolving content of the web.
Methods
The core of the proposed IME system is based on an n-pos model for statistical Kana-Kanji conversion. The authors leverage a massive 2.5TB dataset of Japanese web pages for training. Key methodological contributions include:
- Large-Scale Web Data Training: Utilizing 2.5TB of web pages for training POS tagging, word segmentation, and pronunciation annotation.
- Online Compound Word Mining: A novel algorithm to discover new compound words, collocations, and predicate-argument phrases from web text. This process is designed to be online, allowing for continuous updates.
- Multi-Stage Filtering: A rigorous filtering process to prune the potentially billions of mined entries down to a manageable size for mobile devices. This involves:
- Likelihood ratio method for hypothesis testing.
- LH score for evaluating candidate phrases.
- Usage of cloud service logs to prioritize entries frequently used by users.
- Hand-made deep filtering rules based on POS tags and specific word patterns.
- Cloud-Mobile Hybrid Approach: A 4-gram language model is trained for cloud-based conversion, and its results are interpolated with local mobile device candidates to improve accuracy, especially when wireless connectivity is limited.
Datasets
- Training Data: 2.5TB of Japanese web pages.
- Test Data: 4,978 Kana-Kanji entries from a combination of:
- "JDMWE" (Japanese Dictionary of Multi-Word Expressions) lexicon.
- "Nagoya" compound word lexicon.
- "BCCWJ" (Balanced Corpus of Contemporary Written Japanese) corpus.
- Additionally, a separate experiment used user logs (from two months) for online learning and a test set of 1,248 entries from the latest three days of logs.
Key Insights and Ideas
- The Web as a Living Lexicon: The paper demonstrates that the vast and dynamic nature of the web can be effectively harnessed to build and continuously update language models for IMEs, making them "self-growing."
- Context is King (and Hard to Model): The authors highlight that the correct Kanji selection heavily relies on context, and their work focuses on extracting and utilizing this context efficiently from web data.
- Balancing Size and Accuracy: The core challenge for mobile IMEs is the trade-off between lexicon size and accuracy. The multi-stage filtering process is crucial for achieving high precision with a limited number of entries.
- User Logs as a Gold Standard: Incorporating user logs into the filtering process is a powerful way to align the IME's vocabulary with actual user needs and preferences.
Key Results
- The proposed IME system, when augmented with compound words and cloud service, significantly outperforms two state-of-the-art Japanese IME baselines in terms of top-1, top-6, and top-12 precisions.
- Specifically, the system with cloud integration achieved 88.99% top-1 precision, 94.98% top-6, and 96.44% top-12, with only 41 missing words.
- An experiment testing the "online" ability showed that appending compound entries mined from user logs improved top-1 precision by 3.29% to 5.77%.
- The system has been released for free and is used by millions of users.
Limitations and Future Work
- Resource Constraints: While the filtering process is effective, the paper acknowledges the ongoing challenge of maintaining a good enough local lexicon with an extremely limited number of entries for mobile devices.
- Interpolation Strategy: The paper mentions choosing a simple interpolation strategy for cloud and local candidates (taking the cloud result as the first candidate) and suggests that other methods might offer better comparisons but at a higher network cost.
- Future Work: The paper implies that continuous mining and filtering of web data and user logs will lead to further improvements, making the IME system more adaptive and accurate over time. The "self-growing" nature of the system suggests ongoing development and refinement.
Summary updated