Digital handwriting tests record a pen's position, pressure and timing hundreds of times per second, including while the pen hovers above the tablet between strokes. On small clinical cohorts of roughly 60 to 180 people, they report accuracies near 90% for separating Alzheimer's or Parkinson's patients from healthy controls. Every published number, though, comes out of a single chain of analysis choices. Someone chose how to smooth coordinates, where to set the height at which the pen counts as "lifted", how to normalize pressure, which windows to use for frequency analysis, which subjects to drop for incomplete tasks, and how to split people into train and test sets. None of these chains has been re-run under alternative choices or in a controlled software environment. Neuroimaging has already shown what happens when this is checked. The same MRI scans pushed through three standard pipelines moved test accuracy by up to 9.2%, and models learned pipeline artifacts instead of anatomy (Pipeline-Invariant Representation Learning for Neuroimaging).
We propose to measure how much of a handwriting biomarker's reported accuracy, and of its claimed physiological features, survives variation in defensible analysis choices and in the execution environment. The method is a "multiverse" analysis, meaning the same raw pen data run through every reasonable combination of preprocessing and evaluation choices. We will run it inside hermetic containers: sealed, version-pinned software images, so that any change in results can be traced to an analysis choice and not to library drift. The containers serve as a measuring instrument, not as the contribution. The contribution is a variance decomposition that says whether the field's method comparisons and its headline "in-air" findings are real.
Core Claim
On the NeuroLogical Signals (NLS) and DARWIN-RAW datasets, we expect the spread in accuracy across defensible preprocessing pipelines to be comparable to or larger than the gaps between competing published methods. For example, the gap on NLS clock drawing is 87% for interpretable features with bagging classifiers (Interpretable Features for the Assessment of Neurodegenerative Diseases through Handwriting Analysis) against 90% for velocity-shaded images fed to a ResNet50 (Cross-Corpus and Cross-domain Handwriting Assessment of NeuroDegenerative Diseases via Time-Series-to-Image Conversion). We also expect in-air features to be the most pipeline-sensitive class, because their very definition depends on a hover-detection threshold and on hardware-specific hover sampling. Several papers rank these features as the most discriminative (Consistent Explainable features for Alzheimer's Disease Assessment through Handwriting, How word semantics and phonology affect handwriting of Alzheimer's patients: a machine learning based analysis). If either prediction holds, much of the current literature's ranking of methods and features is unresolved. If both fail, the field gains its first evidence that its results are robust.
Relation to Prior Work
The handwriting papers motivate the question but also leave it open. The explainable-feature study on NLS found 189 of 367 features significant after false-discovery-rate correction, with in-air speed and entropy at $p<10^{-5}$ and $p<10^{-6}$ (Consistent Explainable features for Alzheimer's Disease Assessment through Handwriting). That result comes from one preprocessing chain, and the authors themselves note that not every participant completed every task. The frame-level study sweeps frequency-analysis windows from 25 ms to 1.5 s and reports its best configuration, 89.8% F1 on point tasks for Alzheimer's detection (Detecting Neurodegenerative Diseases using Frame-Level Handwriting Embeddings). A sweep that reports only its best setting is itself a source of optimistic bias that a multiverse analysis makes visible.
A feature taxonomy of more than 160 pen features supplies part of our method, since it fixes feature definitions (A categorisation and implementation of digital pen features for behaviour characterisation). It does not fix the preprocessing upstream of those definitions, and it shipped without a pinned environment. We reuse its definitions and vary what comes before them.
Neuroimaging shapes the design from two directions. The pipeline-invariance work supplies both the finding we test for and a mitigation, multi-pipeline supervised training, which we include as an arm of the study (Pipeline-Invariant Representation Learning for Neuroimaging). That work was done on volumetric MRI with 2,000 subjects; our regime has about 100 subjects, where fitting to analysis choices should be far worse. BIDS Apps supplies the packaging pattern: a standard data layout plus containerized, map-then-reduce execution (BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods). It also imposes a constraint. Containers do not capture the host kernel or hardware floating-point behavior, so environment variance cannot be assumed to be zero and has to be measured across machines. The spike-sorting pipelines, which combine the Nextflow workflow tool with containers (Efficient and reproducible pipelines for spike sorting large-scale electrophysiology data), and Bencher's containerized benchmark services (Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization) supply the orchestration. Each pipeline variant runs as an isolated, re-executable unit, including legacy feature code that needs old library versions.
We borrow one idea from the audit harness for agent-driven science (Audit-Grade Harness for Agent-driven Scientific Computation Workflows): an append-only log of every decision, here every subject exclusion, resampling step and threshold. We deliberately omit its language-model verifier. Our pipelines are deterministic scripts, and a rule-based check of the log is cheaper and cannot hallucinate a verdict. That paper itself flags its verifier as uncalibrated.
Research Program
First, we convert NLS, DARWIN-RAW and PaHaW into one shared layout that stores raw $(x, y, p, t)$ streams, hover samples, device metadata and task labels. Access to NLS and DARWIN-RAW must be negotiated. The Italian word-copying cohort will be added only if its authors release raw data.
Second, we pre-register a pipeline space before looking at any outcomes. It has six axes: coordinate smoothing (none, Savitzky–Golay, low-pass at two cutoffs); hover threshold and minimum pen-up duration; pressure normalization (raw, per-subject z-score, per-device quantile); resampling to a common rate or not; subject-exclusion rule for incomplete tasks; and evaluation protocol (subject-level stratified cross-validation versus nested cross-validation with inner tuning). Each axis gets only options that some published paper has used or that a domain expert would defend. The cross product is sampled at a few hundred points.
Third, we run four model families through every pipeline point: the interpretable-feature bagging classifier, the frame-level spectrogram CNN, the velocity-shaded raster with ResNet50, and a native time-series model (a one-dimensional CNN on the resampled streams). Everything runs in pinned containers, and a subset is replicated on Intel CPUs, AMD CPUs and two GPU generations, both with and without deterministic kernels.
Fourth, we decompose the variance of each test metric $m$ (subject-level AUC or accuracy) as
$$\mathrm{Var}(m) = \sigma^2_{\text{pipeline}} + \sigma^2_{\text{env}} + \sigma^2_{\text{split}} + \sigma^2_{\text{interaction}},$$
where the terms are the variance components due to preprocessing choices, execution environment, random train/test partition of subjects, and their interactions. They are estimated with a crossed random-effects model over the grid. For feature claims, we record for each feature the fraction of pipeline points at which it stays significant after false-discovery-rate correction. This produces a robustness score that can be compared against the published lists.
Baselines
The naive reference point is the published single-pipeline number, taken at face value together with its own confidence interval from split resampling alone. If the pipeline and environment variance components come out small relative to split variance, the naive reference is adequate and our main claim fails.
The strongest existing methods on NLS are the velocity-shaded raster ResNet50, which reports 90% accuracy on Draw Clock and 80.5% on spirals for Alzheimer's detection (Cross-Corpus and Cross-domain Handwriting Assessment of NeuroDegenerative Diseases via Time-Series-to-Image Conversion), and the frame-level CNN at 89.8% F1 on point tasks (Detecting Neurodegenerative Diseases using Frame-Level Handwriting Embeddings). The interpretable bagging model reports 87% on Draw Clock for Alzheimer's versus controls (Interpretable Features for the Assessment of Neurodegenerative Diseases through Handwriting Analysis). On DARWIN-RAW, the reference is CLIP with cross-layer fusion adapters, at an average cross-task AUC of 69.1% (Understanding Cross Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation).
Fairness rules apply to every method. Each receives the same subject splits, the same pipeline grid, the same number of hyperparameter trials tuned inside the inner cross-validation loop, and the same GPU-hour budget. We will first reproduce each published number at its paper's own pipeline point. That establishes whether the published setup can be recovered at all, and discrepancies will be reported as findings. In this study, "beating" a baseline does not mean a higher peak. It means a method whose median across the pipeline grid exceeds the other method's median, with non-overlapping 90% intervals over pipeline points.
The key ablation isolates pipeline diversity during training. Multi-pipeline supervised training, adapted from the neuroimaging work, is compared with single-pipeline training at matched sample count, scored on held-out pipeline points. It tells us whether a model can be made insensitive to preprocessing, or whether variance must instead be removed by standardization.
Why not just…?
Why not just ship a lockfile and fixed seeds instead of containers? For the environment term this may genuinely suffice, and we treat that as a testable outcome. If environment variance across machines is negligible relative to the other terms, we will say so, and conda-level pinning becomes the recommendation. Containers are still needed while running the grid, because the published feature and model code spans incompatible library versions, the same situation Bencher documents (Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization). The BIDS Apps authors also warn that host hardware leaks through regardless, so only a cross-machine replication settles the question.
Why not just report bootstrap confidence intervals over subjects? Bootstrap intervals capture split variance, the third term, and nothing else. The neuroimaging result shows that pipeline variance can be a separate and large component. The decomposition directly tests whether split intervals under-cover: we check whether published numbers fall outside the pipeline spread even when they sit inside their own bootstrap interval.
Why not just train pipeline-invariant models and skip the measurement? This is a real competitor, which is why it appears as the ablation arm. With about 100 subjects, though, invariance training may simply trade pipeline variance for underfitting. Its value also cannot be judged without first knowing how large pipeline variance is. If it closes most of the spread at no cost in median AUC, it becomes the recommended practice, and the multiverse was the measurement that proved it.
Why not just adopt the existing 160-feature library as the standard? It standardizes feature formulas but not the smoothing, hover thresholds or exclusions that feed them. Our per-axis attribution will show whether the upstream choices or the formulas dominate.
Decisive Checks
Three results decide the claim. First, on NLS Draw Clock, is the interquartile range of subject-level accuracy across pipeline points larger than the three-point gap between the raster ResNet50 and bagging methods, and do the method rankings flip on more than a quarter of pipeline points? Second, do in-air speed and entropy stay significant at a lower fraction of pipeline points than on-surface velocity features, and is the hover-threshold axis the one that drives this? Third, simulated hardware shift, made by resampling to lower rates and quantizing pressure to coarser levels, tests whether rasterized and native time-series models degrade differently. This addresses the open question of whether rasterization discards diagnostic high-frequency signal, and answers it inside the same controlled grid instead of across papers with incompatible setups. We will release the grid, containers and decision logs so that any future handwriting biomarker can be reported as a distribution across pipelines, not a single number.
References
- Pipeline-Invariant Representation Learning for Neuroimaging
- Interpretable Features for the Assessment of Neurodegenerative Diseases through Handwriting Analysis
- Cross-Corpus and Cross-domain Handwriting Assessment of NeuroDegenerative Diseases via Time-Series-to-Image Conversion
- Consistent Explainable features for Alzheimer’s Disease Assessment through Handwriting
- How word semantics and phonology affect handwriting of Alzheimer's patients: A machine learning based analysis
- Detecting Neurodegenerative Diseases using Frame-Level Handwriting Embeddings
- A categorisation and implementation of digital pen features for behaviour characterisation
- BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods
- Efficient and reproducible pipelines for spike sorting large-scale electrophysiology data
- Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization
- Audit-Grade Harness for Agent-driven Scientific Computation Workflows
- Understanding Cross Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation