Keyboard Logs as Natural Annotations for Word Segmentation

Fumihiko Takahashi, Shinsuke MoriEMNLP 2015 · DOI

Introduction

In languages such as Japanese and Chinese, text is written without explicit word boundaries (spaces). Consequently, Word Segmentation (WS) is the foundational task for all downstream Natural Language Processing (NLP) applications, including machine translation and sentiment analysis. While current segmenters perform exceptionally well on formal, standard text, their accuracy plummets when encountering "out-of-vocabulary" (OOV) words common in social media, technical domains, or slang. Manually annotating new datasets for every niche domain is economically unfeasible. This paper proposes a novel solution: harvesting "natural annotations" from keyboard input method (IM) logs to provide a continuous stream of domain-specific training data.

Methodology: The IM as a Linguistic Sensor

The researchers treat the Input Method—the software used to convert phonemes (e.g., kana) into characters (e.g., kanji)—as a sensor for human linguistic intent. When a user types a sequence and selects a specific conversion candidate, they are implicitly confirming a word boundary.

To capture this, the authors developed a custom OOV-aware IM server. Unlike standard IMs that rely on fixed dictionaries, this server utilizes a stochastically segmented corpus, allowing it to propose word candidates it has never explicitly seen before based on the probability of character sequences in raw, unannotated text.

The architecture of the custom IM server showing the interaction between client typing and server-side log generation.

Extracting Natural Annotations

The core innovation lies in interpreting user behavior as a labeling task. Every time a user hits a conversion key or the enter key, they provide a snapshot of how a phoneme sequence maps to a segmented word. However, raw logs are inherently noisy, containing typos and fragmented entries.

Examples of raw keyboard logs, showing the mapping from phoneme sequences to edited results and the associated linguistic notes.

To refine these logs into usable training data, the authors introduced two key processing techniques:

  1. Log-chunking: Sequential logs are connected if they occur within a 500ms window, helping to reconstruct the broader sentence context.
  2. Log-mconv (Multiple Conversion Filtering): This filters out "one-off" entries, ensuring the data reflects intentional, successful word selections rather than discarded attempts or errors.

Experimental Results

The framework was evaluated using a Support Vector Machine (SVM) segmenter trained on a combination of high-quality manual data (BCCWJ) and the refined "natural" labels from the IM logs. Testing was conducted on both standard benchmarks and a newly created Twitter evaluation set (TWI-test) consisting of 2,976 manually segmented tweets.

Key Findings:

  • Social Media Performance: On the Twitter dataset, the inclusion of processed IM logs improved the F-measure from 92.14 to 92.90. This represents a significant reduction in error, particularly in recall.
  • OOV Error Mitigation: The ratio of errors specifically caused by out-of-vocabulary words dropped from 23.09% to 17.79%. The logs effectively act as "glue," preventing the model from over-segmenting new entities like location names or slang.
  • Robustness: The addition of noisy log data did not degrade performance on formal text (BCCWJ-test), indicating that the method is stable across domains.

Conclusion and Future Work

This research demonstrates that keyboard logs are a rich, untapped resource for NLP. By leveraging the act of typing as a distributed labeling task, models can adapt to evolving language and niche domains without the need for expensive manual intervention. Future work will focus on scaling the user base to capture more diverse linguistic domains and addressing the privacy considerations inherent in collecting keyboard input data.

Summary updated

BibTeX
@inproceedings{takahashi2015keyboard,
  author = {Fumihiko Takahashi and Shinsuke Mori},
  title = {{Keyboard Logs as Natural Annotations for Word Segmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1140},
}
Figure