Type Anywhere You Want: An Introduction to Invisible Mobile Keyboard

Sahng-Min Yoo, Ue-Hwan Kim, Yewon Hwang, Jong-Hwan KimIJCAI 2021 · DOI

TL;DR

This paper introduces the "Invisible Mobile Keyboard" (IMK), a novel text-entry system for mobile devices that allows users to type anywhere on the screen without predefined key boundaries. They propose a Self-Attention Neural Character Decoder (SA-NCD) that achieves high accuracy in decoding these unconstrained inputs and demonstrate a significant improvement in typing speed and user satisfaction compared to existing methods.

What Problem Does This Paper Address?

Contemporary mobile keyboards, while ubiquitous, suffer from several limitations. The lack of physical feedback leads to increased typos, and the fixed layout of soft keyboards occupies a significant portion of the screen, obstructing content. Furthermore, current systems force a one-size-fits-all approach, ignoring individual users' diverse typing habits and mental models. This paper tackles these issues by proposing an "invisible" keyboard that liberates users from rigid layouts, allowing them to type freely anywhere, while simultaneously addressing the challenges of decoding such unconstrained input.

Methods

The core of the proposed solution is the Self-Attention Neural Character Decoder (SA-NCD). This architecture is designed to decode sequences of touch coordinates into text. It comprises two main modules:

  • Geometric Decoder: This module uses a Bidirectional Gated Recurrent Unit (BiGRU) to process the raw touch input sequence and generate an initial character prediction. BiGRUs are employed to capture temporal dependencies in the touch data and overcome the vanishing gradient problem.
  • Semantic Decoder: This module, based on the Transformer architecture, takes the output from the geometric decoder and refines it. It uses a self-attention mechanism to consider semantic context and correct errors, including those arising from the geometric decoder's initial predictions or user typos.

A crucial component is Confidence Masking. This process identifies low-confidence predictions from the geometric decoder and replaces them with special mask tokens before feeding them to the semantic decoder. This allows the semantic decoder to focus on correcting potentially erroneous or ambiguous inputs. The system also incorporates a novel training scheme involving pre-training the semantic decoder on a large text corpus (Billion Word Benchmark) and then fine-tuning both modules together.

Datasets

The paper introduces a newly collected, large-scale, and richly annotated IMK dataset. This dataset contains approximately 2 million pairs of touch points and corresponding characters, collected from 88 participants. This dataset is significantly larger than those used in previous text-entry research. For pre-training the semantic decoder, they utilized the Billion Word Benchmark (1BW). The MacKenzie (MacK) and Common English phrases sets were used for evaluation.

Key Insights and Ideas

  • Liberating Typing: The fundamental idea of an "invisible" keyboard that allows typing anywhere is a significant departure from current paradigms, promising a more intuitive and less restrictive user experience.
  • Geometric + Semantic Decoding: The two-stage decoding approach (geometric for initial prediction, semantic for refinement) effectively addresses the dual challenges of interpreting raw touch data and understanding linguistic context.
  • Importance of Pre-training: Pre-training the geometric decoder on raw touch data and the semantic decoder on a large text corpus is critical for achieving high accuracy, highlighting the value of leveraging existing large datasets.
  • User Variability is Key: The analysis of user behavior reveals significant individual differences in typing patterns, underscoring the need for adaptive or unconstrained input methods like IMK.
  • Confidence Masking as a Bridge: The confidence masking mechanism acts as an intelligent gatekeeper, ensuring that the semantic decoder receives the most informative input from the geometric decoder.

Key Results

  • The proposed SA-NCD achieved a Character Error Rate (CER) of 5.51% on the MacKenzie set and 7.71% on the Common set, which consisted of unseen phrases from unknown users. This performance is significantly better than other sequential models tested.
  • A user study demonstrated that participants could type 51.6 WPM using the IMK with the SA-NCD decoder, which is 157.5% faster than using third-party soft keyboards on their own smartphones.
  • The IMK system also surpassed other text entry methods, including gesture-based and touch-based methods, in terms of Words Per Minute (WPM).
  • User feedback indicated low physical, mental, and temporal demand, along with high satisfaction when using the IMK.

Limitations and Future Work

  • Data Collection: While the dataset is large, it's noted that the data was collected using a web interface, and further research into real-world mobile application integration is needed.
  • Front-end Design: The paper suggests that future work should focus on a more polished front-end design for practical implementation of the IMK.
  • Generalizability: While the system performed well on unseen phrases and users, further validation across a wider range of user demographics and typing styles would be beneficial.
  • Real-time Performance: Although the results are promising, optimizing the decoder for even faster real-time performance on resource-constrained mobile devices remains an area for improvement.

Summary updated

BibTeX
@inproceedings{yoo2021type,
  author = {Sahng-Min Yoo and Ue-Hwan Kim and Yewon Hwang and Jong-Hwan Kim},
  title = {{Type Anywhere You Want: An Introduction to Invisible Mobile Keyboard}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/242},
}

Loading related content...

Figure