Handwriting Prediction Considering Inter-Class Bifurcation Structures

Masaki Yamagata, Hideaki Hayashi, Seiichi Uchida

Introduction

Predicting the future trajectory of handwriting is a complex task in time-series analysis. Unlike simple physical systems, handwriting is driven by human intent, which is often ambiguous in its early stages. A significant challenge in this domain is inter-class bifurcation: the phenomenon where different characters (e.g., '0' and '6') share nearly identical initial strokes. Standard predictive models often fail here, either by averaging the possible paths into a nonsensical "middle ground" or by prematurely committing to a single, potentially incorrect, class.

The Class-Guided Prediction (CGP) Model

To address this, the authors propose the Class-Guided Prediction (CGP) model. The core innovation lies in factorizing the prediction task into two distinct components: class probability and class-conditional coordinate prediction.

Figure 1: The CGP architecture uses a three-layer LSTM with skip connections to feed dual output modules for class probability and class-specific GMM parameters.

As shown in Figure 1, the model extracts features from a sequence of relative pen coordinates using a 3-layer LSTM with skip connections. These features are then processed by two parallel modules:

  1. Class Probability Prediction: Outputs a categorical distribution representing the likelihood of each digit class (0-9).
  2. Coordinate Prediction: Outputs parameters for a Gaussian Mixture Model (GMM) for each class, defining the specific trajectory dynamics for that character.

By training these components jointly using a combined loss function (negative log-likelihood and cross-entropy), the model learns to represent multiple distinct futures simultaneously. This prevents "mode collapse," where a model merges diverging paths into a single, high-variance blob.

Handling Ambiguity and Bifurcation

The CGP model treats uncertainty as a structural feature rather than a bug. Early in a stroke, when the writer's intent is unclear, the class probabilities are distributed across several candidates. Consequently, the sampled future trajectories branch out, representing all likely characters. As the pen moves and the ambiguity is resolved, the class probabilities shift toward the true character, and the predicted trajectories naturally collapse into a single, accurate path.

Figure 5: Evolution of predicted trajectories over time. As more of the stroke is observed, the model narrows its hypotheses, eventually converging on the correct character path.

Quantitative analysis using the UNIPEN database confirms this behavior. The authors observed that prediction error (RMSE) tends to spike at bifurcation points—where paths for different digits diverge—before dropping once the class becomes identifiable. In comparative tests, CGP significantly outperformed deterministic LSTMs and standard MDNs in long-term prediction accuracy, particularly because it maintains distinct modes for different classes rather than blurring them.

Conclusion

The CGP framework provides a robust method for handwriting prediction by explicitly modeling the underlying class structure. By separating "what is being written" from "how it is written," the model handles inter-class bifurcations gracefully. Future work may extend this to continuous handwriting or leverage the bifurcation detection for ultra-fast character recognition, identifying a digit the moment its trajectory becomes unique.

Summary updated

Abstract

Temporal prediction is a still difficult task due to the chaotic behavior, non-Markovian characteristics, and nonstationary noise of temporal signals. Handwriting prediction is also challenging because of uncertainty arising from inter-class bifurcation structures, in addition to the above problems. For example, the classes `0' and `6' are very similar in terms of their beginning parts; therefore it is nearly impossible to predict their subsequent parts from the beginning part. In other words, `0' and `6' have a bifurcation structure due to ambiguity between classes, and we cannot make a long-term prediction in this context. In this paper, we propose a temporal prediction model that can deal with this bifurcation structure. Specifically, the proposed model learns the bifurcation structure explicitly as a Gaussian mixture model (GMM) for each class as well as the posterior probability of the classes. The final result of prediction is represented as the weighted sum of GMMs using the class probabilities as weights. When multiple classes have large weights, the model can handle a bifurcation and thus avoid an inaccurate prediction. The proposed model is formulated as a neural network including long short-term memories and is thus trained in an end-to-end manner. The proposed model was evaluated on the UNIPEN online handwritten character dataset, and the results show that the model can catch and deal with the bifurcation structures.
BibTeX
@inproceedings{yamagata2020handwriting,
  author = {Masaki Yamagata and Hideaki Hayashi and Seiichi Uchida},
  title = {{Handwriting Prediction Considering Inter-Class Bifurcation Structures}},
  booktitle = {ICFHR},
  year = {2020},
  doi = {10.1109/icfhr2020.2020.00029},
}
Figure