Digital pens and tablets record handwriting as a time series: pen position, pressure, and whether the pen is touching the surface or hovering, sampled hundreds of times per second. These recordings carry signs of Alzheimer's and Parkinson's disease that are invisible in the finished drawing, such as slower loops, longer hesitations, and tremor. Clinical datasets are tiny, though. The main Alzheimer's handwriting benchmark has 174 people. What these datasets do have is many recordings per person: each participant performs a battery of 14 to 25 different writing and drawing tasks. Current models either train one classifier per task, or squash the trajectories into images so they can borrow a large pretrained vision model. Both approaches treat the battery as a set of separate problems. Both also use only one bit of supervision per person: impaired or not.
We propose a different use of the battery. A single temporal encoder is trained on all tasks with two goals. The first is the usual diagnosis label. The second is an auxiliary goal: tell whether two recordings from different tasks came from the same person. Matching a person's spiral to the same person's word copy cannot be done from task-specific shape. It forces the encoder to represent what stays constant about that person's motor system across tasks. Disease is a property of the person, so this is plausibly the right thing to represent. The auxiliary goal turns 174 binary labels into a 174-way identity problem with thousands of cross-task pairs. Our central bet is that this cross-task identity signal regularizes small-cohort screening better than the obvious auxiliary, which is reconstructing or forecasting the trajectory. It should help most on tasks the model never saw with diagnostic labels.
Core Claim
In clinical handwriting cohorts with few subjects but many tasks per subject, the bottleneck is supervision about the person rather than gradient conflict between tasks or a lack of pretrained capacity. A 1D temporal encoder trained jointly on diagnosis and cross-task writer matching should beat three alternatives: (a) per-task classifiers fused by voting, (b) jointly trained supervised models with or without gradient balancing, and (c) the same encoder with a trajectory-forecasting auxiliary. The advantage should be clearest under held-out-task evaluation and should grow as the cohort shrinks. If writer matching gives no gain over (a) and (c), the claim is wrong.
Where This Sits in Prior Work
Three lines of work motivate the gap. Hybrid Transformer for Early Alzheimer's Detection combines stroke images with 1D kinematic signals on DARWIN-RAW and adds a template contrastive loss, which pulls each class toward a learned prototype. It reports its best result on one isolated task, writing the letter 'L'. Understanding Cross Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation renders trajectories as RGB images, adapts a frozen CLIP model with small adapter modules, and tests on unseen tasks. It finds word copying transfers best and reports a mean cross-task AUC of 69.1%. Detecting Neurodegenerative Diseases using Frame-Level Handwriting Embeddings turns signals into short-time spectrograms and trains a separate CNN per task and per disease pair on the Johns Hopkins NLS cohort.
All three use only diagnosis labels and treat tasks independently or sequentially. Our proposal keeps raw temporal sequences, trains one encoder across the whole battery, and adds subject-identity supervision that none of them use. The modest 69.1% cross-task AUC is the evidence that task-specific features do not carry over. Our auxiliary goal is designed to fix that failure directly.
Two papers supply methods we reuse or test against. Joint Classification and Trajectory Regression of Online Handwriting using a Multi-Task Learning Approach shows that a shared 1D CNN trunk, split late into a classification head and a trajectory-regression head, gives smoother reconstructions than regression alone. That study used healthy writers and reconstructed the trajectory from IMU signals. On clinical tablet data the trajectory is already the input, so reconstruction becomes an autoencoding task. That is the reason we treat trajectory forecasting as a competitor rather than as our core. RotoGrad attributes negative transfer to tasks whose gradients differ in size or point in opposite directions, and corrects both. We use it to test the alternative explanation that joint training on clinical batteries fails because of gradient conflict.
Some work constrains the design. Assessing the Potential of Data Augmentation in EEG Functional Connectivity for Early Detection of Alzheimer's Disease found that recombining signal components helped spatial models but hurt a purely temporal model. We therefore avoid synthetic recombination of trajectories. Towards AI-driven Next Generation Personalized Healthcare and Well-being warns that small clinical models memorize acquisition artifacts. This is the main threat to writer matching, since the encoder could match people by session-specific tablet or pressure quirks rather than motor traits. We address it head on below.
How word semantics and phonology affect handwriting of Alzheimer's patients shows that hovering features carry planning deficits. We therefore keep pen-up samples in the input rather than discarding them.
Research Program
The encoder is a small non-causal temporal convolutional network, meaning a 1D convolutional network that looks both backward and forward in time, in the style of Online handwriting trajectory reconstruction from kinematic sensors using temporal convolutional network. Its input per time step is position increments, pressure, pen state, and velocity. The network sees a learned task embedding that rescales its intermediate features, so one set of weights can serve all tasks. Its output is one vector per recording.
To remove obvious session cues, each recording is normalized before encoding: absolute position is subtracted, scale is normalized, and pressure is z-scored within the recording.
Training batches sample $K$ subjects and $M$ different tasks per subject. The loss is
$$\mathcal{L} = \mathcal{L}_{\text{diag}} + \lambda\, \mathcal{L}_{\text{match}},$$
where $\mathcal{L}_{\text{diag}}$ is binary cross-entropy for the impaired-versus-control label on every recording. $\mathcal{L}_{\text{match}}$ is a contrastive loss in which two recordings of different tasks from the same subject are a positive pair and recordings from other subjects in the batch are negatives. The weight $\lambda$ sets the balance between the two.
Same-task pairs are never used as positives, so the encoder cannot match people through task-specific shape. At test time all subjects are unseen. A subject-level score is the average of per-recording diagnosis probabilities over whatever tasks that subject performed.
The analysis has three parts. First, a cohort-size curve that subsamples 40, 80, and all 174 DARWIN subjects. Regularization from extra supervision should matter most when labels are scarcest. Second, a probe of what the matching embedding encodes. We correlate it with interpretable person-level quantities, namely the fraction of time spent hovering, mean loop speed, and tremor-band spectral power. We also test whether a linear classifier on the embedding separates patients from controls when trained with zero diagnostic gradient ($\lambda$ only). Third, an artifact audit. We train a matcher on session-only features (sampling jitter, pressure offset, recording start position) and check that it cannot reach the matching accuracy of the full model. On NLS, where control, Parkinson's, Alzheimer's, and Parkinson-mimic groups coexist, we also check whether identity features cluster by diagnosis or merely by demographics.
Baselines
The naive reference point is a gradient-boosted classifier on handcrafted summary features per task: duration, mean speed, pressure statistics, and in-air time. Task-level predictions are averaged per subject. Predicting cognitive impairment using novel functional features of spatial proximity and circularity in the digital clock drawing test found such summary features matched functional curves at an AUC of 0.90 on the large Framingham clock-drawing cohort, so this floor may not be low. No reported result was found on Lacuna for a summary-feature ensemble on DARWIN-RAW. We will produce one.
The strongest existing method on the cross-task question is the CLIP adapter approach (CLFA) of Understanding Cross Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation, which reports a mean cross-task AUC of 69.1% on DARWIN-RAW. We will rerun it on our exact subject folds and task splits rather than compare to the published figure, and give it the same hyperparameter search budget. Beating it means a higher mean held-out-task AUC whose subject-level bootstrap interval excludes CLFA's.
On within-task accuracy, Hybrid Transformer for Early Alzheimer's Detection reports 90.91% accuracy and 90.32% F1 on DARWIN Task 8. We will reproduce it on our folds as the single-task ceiling. We do not need to beat it on Task 8. We need to match it there while generalizing better.
On NLS, the per-task spectrogram CNNs of Detecting Neurodegenerative Diseases using Frame-Level Handwriting Embeddings report 89.8% F1 for Alzheimer's detection on Point Tasks and 74.5% F1 for Parkinson's on CopyReadText. These are the per-task numbers to match.
Three ablations isolate the key ingredient:
- Same encoder with $\lambda=0$, trained jointly on diagnosis only.
- The same diagnosis-only joint model with RotoGrad applied across task groups.
- The same encoder with $\mathcal{L}_{\text{match}}$ replaced by an auxiliary that forecasts the next 200 ms of trajectory and the duration of the next pen lift. This is the Ott-style regularizer adapted to clinical data, with its weight tuned on the same grid.
All neural models share parameter count, epochs, augmentation, and a fixed number of tuning trials selected by inner cross-validation on subjects only.
Why not just…?
Why not just train per-task classifiers and vote across the battery? This is a real threat for subjects who completed every task, since late fusion averages out per-task noise. It cannot score a task it never trained on, however, and it learns nothing shared across tasks. The held-out-task protocol settles the comparison. Within-battery, if voting matches our model at full cohort size but loses at 40 and 80 subjects, the regularization claim holds. If voting wins everywhere, we report that writer matching only buys task transfer.
Why not just fix negative transfer with gradient balancing? The gap analysis in the literature suggests joint models across clinical batteries fail because of conflicting gradients, and RotoGrad directly addresses that. However, every task in a subject shares that subject's label, so the tasks are closer to multiple views of one person than to competing objectives. We expect little directional conflict. Ablation 2 tests this. If RotoGrad alone closes the gap, the problem was optimization and our claim is wrong.
Why not just add trajectory reconstruction as the auxiliary, as in joint classification-regression work? Joint Classification and Trajectory Regression of Online Handwriting using a Multi-Task Learning Approach shows the benefit when reconstruction crosses modalities, from IMU to position. On tablet data the target is the input, and forecasting can be solved largely from local smoothness without any person-level feature. Still, forecasting pen-lift timing might capture the planning deficits highlighted by How word semantics and phonology affect handwriting of Alzheimer's patients, so it is a credible rival. Ablation 3 compares the two directly, and we also run both together in case they are complementary.
Why not just use a pretrained vision model on rendered strokes? This is the CLFA route. Its 69.1% cross-task AUC is the evidence that frozen image features plus adapters do not transfer well across tasks. Rendering also discards timing and hover information that Dynamics of Handwriting for Cognitive Assessment found discriminative when the drawings looked visually identical. It remains our headline baseline.
Proposed Experiments and Decisive Checks
All evaluation uses subject-disjoint, repeated nested cross-validation with subject-level bootstrap intervals. With 174 subjects, differences of a few AUC points fall within noise, and we will say so instead of claiming them.
On DARWIN-RAW we follow the cross-task protocol of Understanding Cross Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation: train on a subset of task types and report AUC per unseen task and averaged. We also report full-battery subject-level AUC and accuracy. On NLS we report per-task F1 for Alzheimer's and Parkinson's detection plus a four-way subject-level macro F1. We train writer matching across all 14 tasks and use the Parkinson-mimic group to check that identity features do not merely encode "has a movement disorder."
Three results decide the proposal. The first is held-out-task AUC: the full model versus $\lambda=0$, versus the forecasting auxiliary, and versus CLFA. The second is the cohort-size curve, where the gap over $\lambda=0$ should widen as subjects are removed. The third is the artifact audit, where a session-feature matcher should fall well short of the learned matcher's accuracy. Writer matching should also stop improving diagnosis when matching is restricted to tasks recorded in different sessions, where available.
If the first two hold and the third passes, cross-task identity is a usable source of supervision for small clinical cohorts. If the third fails, the gains reflect acquisition leakage and the method should not be used clinically.
References
- Hybrid Transformer for Early Alzheimer's Detection: Integration of Handwriting-Based 2D Images and 1D Signal Features
- Understanding Cross Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation
- Detecting Neurodegenerative Diseases using Frame-Level Handwriting Embeddings
- Joint Classification and Trajectory Regression of Online Handwriting using a Multi-Task Learning Approach
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Assessing the Potential of Data Augmentation in EEG Functional Connectivity for Early Detection of Alzheimer's Disease
- Towards AI-driven Next Generation Personalized Healthcare and Well-being
- How word semantics and phonology affect handwriting of Alzheimer's patients: A machine learning based analysis
- Online handwriting trajectory reconstruction from kinematic sensors using temporal convolutional network
- Predicting cognitive impairment using novel functional features of spatial proximity and circularity in the digital clock drawing test
- Dynamics of Handwriting for Cognitive Assessment