Decoding imaginary handwriting trajectories with shape and time distortion loss for brain-to-text communication

Xu Guang-xiang, Zebin Wang, Kedi Xu, Junming Zhu, Jianmin Zhang, Yueming Wang, Yaoyao HaomedRxiv 2024 · DOI

TL;DR

Researchers have developed a new brain-computer interface (BCI) framework that reconstructs complex, imaginary handwriting trajectories—such as Chinese characters—directly from neural signals. By replacing standard training metrics with a specialized "DILATE" loss function that accounts for timing and shape misalignments, the system achieved a 91.1% recognition rate across a 1,000-character database, offering a path toward universal, language-agnostic brain-to-text communication.

What Problem Does This Paper Address?

Current high-performance "brain-to-text" BCIs often rely on classification: the computer tries to guess which of the 26 English letters a user is imagining. While effective for Latin-based languages, this approach hits a wall with languages like Chinese, which contain thousands of unique characters.

To create a universal system, we need trajectory reconstruction—literally "drawing" the intended shape from brain activity. However, a major hurdle exists: paralyzed patients cannot provide "ground truth" movement data. They follow a visual guide, but their "thoughts of motion" often lag behind or lead the prompt. Standard machine learning models use Mean Squared Error (MSE) loss, which demands perfect point-to-point alignment. If a user imagines a stroke perfectly but starts 200ms late, MSE treats it as a total failure. This misalignment leads to blurry, unrecognizable decodings.

Methods

The authors recorded intracortical signals from two Utah microelectrode arrays implanted in the motor cortex of a paralyzed patient. To solve the misalignment issue, they introduced a decoding framework using DILATE (DIstortion Loss including shApe and TimE).

Unlike MSE, DILATE consists of two parts:

  1. Shape Loss: Based on "soft" Dynamic Time Warping (DTW), it allows the model to "stretch" or "compress" the predicted trajectory in time to find the best structural match with the target.
  2. Time Loss: A penalty term that prevents the model from drifting too far from the actual timeline.

The team used a Long Short-Term Memory (LSTM) network to translate neural features (specifically Entire Spiking Activity, or ESA) into velocity profiles. These profiles were then integrated to reconstruct the character's path and matched against a library of standard characters using DTW-based template matching.

Datasets

  • Primary Dataset: 180 Chinese characters recorded over 6 sessions from a 70-year-old patient with a C4-level spinal cord injury.
  • Expanded Library: A 1,000-character database used to test the robustness of the recognition algorithm.
  • Benchmark Dataset: A publicly available dataset of English letter handwriting (from the seminal Willett et al., 2021 study) was used to validate that the DILATE approach generalizes across different languages and subjects.

Key Insights and Ideas

  • Shape Over Synchronicity: In handwriting, the "gestalt" (the overall form) is far more important for recognition than whether every pixel was drawn at the exact millisecond. DILATE prioritizes this structural integrity.
  • Handling "Imagery Jitter": The study found that misalignment isn't uniform; a patient might speed up during an easy stroke and slow down during a complex one. DILATE’s ability to handle non-linear temporal shifts is what makes it superior to MSE in clinical settings.
  • The Multi-Day Advantage: Typically, BCIs struggle when combining data from different days because neural signals "drift." However, the authors found that DILATE actually thrives on multi-day data fusion, yielding a 15% performance boost where MSE-based models typically plateau.

Key Results

  • High Accuracy: The system achieved a 91.1% recognition rate for Chinese characters within a 1,000-character library.
  • Outperforming the Standard: DILATE improved recognition rates by 10% over MSE for Chinese characters and 13.5% for English letters.
  • Human-Readable Output: Even without computer assistance, the trajectories reconstructed via DILATE were visually recognizable as characters, whereas MSE-reconstructed paths were often distorted or "flat-lined" due to timing mismatches.
  • Single-Trial Success: The method proved capable of high-fidelity reconstruction on a single-trial basis, moving away from the need to average multiple attempts to get a clear signal.

Limitations and Future Work

While a significant leap forward, the authors acknowledge several areas for growth:

  • Offline vs. Online: This study was conducted "offline" (analyzing recorded data). The next step is "online" testing, where the patient receives real-time visual feedback of their reconstructed handwriting.
  • 2D vs. 3D: The current model treats handwriting as a 2D plane movement. Future versions could incorporate 3D dynamics to better capture the nuances of neural motor intent.
  • Communication Speed: While the recognition is highly accurate, the paper does not focus heavily on the "characters per minute" speed, which is a critical metric for practical communication tools.

Summary updated

Abstract

Abstract The potential to decode handwriting trajectories from brain signals has yet to be fully explored in clinical brain-computer interfaces (BCIs). One of challenges remains that the clinical BCIs mostly rely on imaginary movement due to motor deficit of the subject, which often leads to misalignment with neural activity and impedes accurate decoding. Here, we recorded intracortical neural signals from a paralyzed patient during imaginary handwriting of Chinese characters, from which the trajectories of handwriting were reconstructed and translated into texts using machine learning approach. We introduced an innovated decoding framework that incorporates a novel loss function, DILATE, to accommodate both shape and temporal distortions between movement and neural activity to account for the misalignment issue. Our method reconstructed closely resembled and human-recognizable handwriting trajectories, outperforming the conventional mean square error loss by 10% of recognition rate. Moreover, the new decoding framework enabled effective multi-day data fusion, resulting in further 15% enhancement. With a dynamic time warping approach, the recognition rate achieved up to 91.1% within a 1000-character database. Additionally, we applied our method to a previous subject who imaged handwriting of English letters, showcasing its capability for single-trail trajectory reconstruction and 13.5% higher recognition outcomes. Altogether, these findings demonstrated a new decoding scheme for BCIs that could accurately reconstruct the imaginary handwriting trajectory. This advancement paves the way for a universal brain-to-text communication system that is applicable to any written language, marking a significant leap forward in the field of neural decoding and BCI technology.
BibTeX
@misc{guangxiang2024decoding,
  author = {XU Guang-xiang and Zebin Wang and Kedi Xu and Junming Zhu and Jianmin Zhang and Yueming Wang and Yaoyao Hao},
  title = {{Decoding imaginary handwriting trajectories with shape and time distortion loss for brain-to-text communication}},
  year = {2024},
  doi = {10.1101/2024.07.02.24309802},
  howpublished = {medRxiv},
}
Figure