Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen

Mohamad Wehbi, Daniel Luge, Tim Hamann, Jens Barth, Peter Kämpf, Dario Zanca, Bjoern M. Eskofier

TL;DR

This paper introduces a novel approach for reconstructing handwritten trajectories and recognizing words using an IMU-enhanced digital pen, eliminating the need for specialized writing surfaces. By training a deep learning model to directly map raw IMU sensor data to relative pen tip displacements, they achieve accurate surface-free trajectory reconstruction and enable word recognition, paving the way for more versatile digital handwriting applications.

What Problem Does This Paper Address?

Digitizing handwriting is crucial for various applications, from note-taking to document signing. While handwriting recognition systems have advanced, reconstructing the actual pen trajectory (not just converting to text) has been challenging, especially for multi-stroke words. Existing methods often rely on specific writing surfaces (like tablets) or complex sensor fusion techniques that are prone to error accumulation, limiting them to single strokes or requiring extensive pre-processing. This paper tackles the problem of surface-free, multi-stroke trajectory reconstruction and word recognition using a single, IMU-enhanced digital pen.

Methods

The core of this work is a data-driven approach using a Convolutional Neural Network (CNN). Instead of relying on traditional sensor fusion, the CNN is trained to directly map raw IMU sensor data (accelerometer, gyroscope) to relative displacement vectors of the pen tip.

Key methodological aspects include:

  • Surface-Free Approach: The system is designed to work without requiring a specific writing surface, allowing users to write on regular paper.
  • IMU-Enhanced Digital Pen: A STABILO Digipen, equipped with IMUs, is used to capture motion data.
  • Data Synchronization and Preparation: A crucial step involves synchronizing data from the pen (100 Hz) and the tablet (60 Hz). They explored three pre-processing methods:
  • Label Interpolation: Upsampling tablet data to match pen data length.
  • Stroke Interpolation: A more refined method that synchronizes based on touch/hover strokes, aiming to better align force and pressure data.
  • Chunking: Dividing data into fixed-length segments (10 timesteps) to simulate real-time prediction.
  • CNN Architecture: A one-dimensional CNN with multiple convolutional layers, batch normalization, and dropout is used for regression, mapping IMU data to X and Y displacement coordinates.
  • Trajectory Reconstruction: Predicted displacement vectors are cumulatively summed to reconstruct the absolute trajectory.
  • Word Recognition: A separate recognition model (CLDNN) is trained on the reconstructed trajectories to recognize words. A joint model combining IMU and trajectory data is also explored.

Datasets

  • Custom Dataset: Data was collected using three STABILO Digipens and a Samsung Galaxy Tab S6. Six users wrote random words displayed on the tablet. This dataset includes IMU sensor data, tablet positional data (ground truth), and word text labels.
  • Existing Dataset [32]: A dataset of 27,961 words written on plain paper using a Digipen was used to evaluate the system's performance on paper-written data via text recognition.

Key Insights and Ideas

  • Direct Mapping for TR: The success of directly mapping raw IMU data to displacement vectors via a CNN is a significant departure from traditional sensor fusion methods, simplifying the pipeline and potentially reducing error accumulation.
  • Stroke Interpolation's Importance: Differentiating between touch and hover strokes during data preparation proved crucial for generating smoother, more accurate trajectories, especially for curved elements. Label interpolation led to more "straight" reconstructions.
  • Surface Independence is Achievable: The system demonstrates that accurate trajectory reconstruction is possible without relying on specialized writing surfaces, opening up new possibilities for digital pen applications.
  • Joint IMU-Trajectory Recognition Boost: Combining IMU data with the reconstructed trajectories in a recognition model yields better results than using IMU data alone, highlighting the quality of the generated trajectories.

Key Results

  • Trajectory Reconstruction: The stroke interpolation method achieved a normalized error rate of 0.1762 on tablet-written data, outperforming label interpolation (0.1864). Chunking also showed competitive results (0.1772).
  • Word Recognition (Paper Data):
    • The baseline IMU-only model achieved a Character Error Rate (CER) of 16.0683%.
  • Models trained on generated trajectories performed worse than the IMU-only baseline:
  • Label interpolation: 21.0789% CER * Stroke interpolation: 19.8676% CER * Chunking: 19.5103% CER
    • The joint model (IMU + trajectory data) achieved the best recognition rate with a CER of 15.3141%, outperforming the IMU-only baseline by 0.7542%.

Limitations and Future Work

  • Data Limitations: The dataset primarily consisted of Latin alphabet words written left-to-right. This led to some limitations, such as the system learning a constant rightward movement even during hovering, and characters not always aligning perfectly on a straight line. The ratio of touch to hover strokes (68% touch, 32% hover) also impacted performance on hover strokes.
  • Overfitting: The models showed lower error rates on training data compared to test data, suggesting potential overfitting.
  • Future Work:
    • More Diverse Data: Collecting data with more hovering, balanced touch/hover ratios, and varied movement patterns (not just words) is suggested to improve robustness.
  • Extended Reconstruction: Reconstructing complete sentences and random images, not just words.
  • Generalization: Expanding beyond the Latin alphabet to multi-lingual trajectory reconstruction.
  • Alternative Architectures: Exploring architectures like Autoencoders for trajectory retracing.
  • Real-time Performance: Further optimization for live-streamed writing sessions.

Summary updated

BibTeX
@inproceedings{wehbi2022surface,
  author = {Mohamad Wehbi and Daniel Luge and Tim Hamann and Jens Barth and Peter Kämpf and Dario Zanca and Bjoern M. Eskofier},
  title = {{Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=j8JzHKnzMvk},
}
Figure