Do Video Models See the Sequence Effect? Cycle-Aligned Tokens for Bradykinesia

In Parkinson's disease, slowness of movement (bradykinesia) is not just "slow." Clinicians rating a ten-second finger-tapping test look for the sequence effect: taps that start reasonably large and fast, then shrink and space out as the trial goes on. That decline is a relation between early and late movement cycles. It cannot be read off any single second of video. Video networks built from small sliding filters see only a few frames at a time. Global-attention models such as TimeSformer can in principle see the whole clip, but they cut it into fixed-length time chunks and average the result into one vector. Nobody has checked whether either kind of model actually uses the order of movement cycles when it predicts a severity score. The proposal has two parts. The first is a cheap diagnostic: shuffle the order of the movement cycles in a video, keeping each cycle intact, and measure how much the model's prediction moves. The second is a model whose tokens (the units the model processes) are individual movement cycles rather than fixed time windows, with separate output heads for per-cycle amplitude and per-cycle interval.

The motor-slowness literature gives a second reason to care. Granular analysis of finger tapping found that amplitude decrement and interval prolongation behave as distinct phenomena, possibly with different neural origins (Zarrat Ehsan et al.). Imaging work on slowness likewise points to separable network causes rather than one "motor output" failure. These include hyperactive subthalamic–striatal connectivity that tracks symptom state (Yin et al.) and upstream visual-to-motor disconnection between area MT+ and primary motor cortex (Song et al.). A video model that returns one ordinal score merges these components back together. A model that outputs separate amplitude and interval trajectories, cycle by cycle, gives a measurement that questions about separate causes can actually be tested against.

Core Claim

For bradykinesia, the receptive field that matters is reach across movement cycles, not reach across pixels or seconds. We claim three things. First, current video models, both local and global, are largely insensitive to cycle order, so they score the sequence effect through correlated local statistics such as mean amplitude. Second, a model that tokenizes by detected movement cycle and must predict per-cycle amplitude and interval will be order-sensitive by construction. Third, that model will match handcrafted keypoint pipelines where hand landmarks are tracked reliably and beat them where landmark tracking fails. The first claim is falsifiable with a shuffle test that needs no new model. The third claim is the one most likely to fail, and we treat it that way.

Relation to Prior Work

Three existing works come closest, and each leaves a different piece open.

GTSN addresses local temporal receptive fields in Parkinson's video. Its Global Shift Module lets every temporal segment exchange features with every other segment, and it amplifies tiny motions with Eulerian Video Magnification. It targets tremor, however, uses fixed temporal segments, and never tests whether order matters. It motivates our work because it shows that global temporal exchange beats a local temporal shift module. We differ in aligning tokens to movement cycles and in probing order dependence directly.

Banerjee et al. benchmark frozen video foundation models, meaning large pretrained video networks used without further training, with a linear head across 16 clinical tasks. They find TimeSformer competitive on finger tapping and sensitivity of only 43.2–57.3%. This constrains the design, because global attention alone is clearly not enough. It also leaves open whether the missed cases come from features that ignore order.

Zarrat Ehsan et al. supply our decomposition. They extract MediaPipe hand landmarks and compute 13 interpretable features, including regression slopes of amplitude and interval over the trial. Those features see the whole trial trivially. Their weakness is landmark failure under occlusion and fast movement, which is also the weakness we must show we overcome.

On the architecture side, CCNet supplies a technique we borrow. It factorizes global attention along orthogonal axes to cut memory, and we factorize analogously into a within-cycle encoder and an across-cycle sequence model. EccoMamba shows that flattening structured data into one-dimensional sequences introduces scan-order artifacts. It serves as a caution against using a state-space model over raw frames. Movement cycles give a natural one-dimensional order in which scanning direction is meaningful rather than arbitrary.

Research Program

The work proceeds in three stages.

The first stage is the diagnostic. Cycle boundaries are found by a landmark-free repetition detector. It computes a temporal self-similarity matrix of low-level frame features, which scores how alike every pair of frames looks, and peak-picks the period. Where landmarks are reliable, we check these boundaries against MediaPipe fingertip-distance minima. We then build cycle-shuffled versions of each test video, with a short crossfade at each splice. We also build a control that shuffles within-cycle frame order, which destroys local dynamics but keeps cycle order. For any trained scorer, order sensitivity is the mean absolute change in its predicted severity under cycle shuffling, normalized by the change under an equal number of random splice cuts that preserve order. The normalization separates real order use from sensitivity to splice artifacts. A scorer that uses the sequence effect should shift predictions toward lower severity when a decaying trial is shuffled, because shuffling removes the systematic decline.

The second stage is the model. Each detected cycle is encoded at full frame rate and native hand-crop resolution by a video encoder initialized from TimeSformer or V-JEPA2 weights and fine-tuned. The encoder sees only frames inside that cycle, which protects the fine spatial detail needed for millimetre-scale amplitude changes. A small transformer with learned cycle-index embeddings then runs over the sequence of cycle tokens. The model has three heads. The first predicts the MDS-UPDRS item score with the hybrid ordinal-focal loss of Lu et al.. The other two regress per-cycle amplitude and per-cycle interval. The auxiliary targets come from MediaPipe on frames where landmark confidence is high and are masked elsewhere, so the model learns those quantities from pixels and can produce them where landmarks fail.

The third stage connects the outputs to etiology. The personalized Parkinson project finger-tapping dataset covers several medication states. We test whether the model's amplitude-slope and interval-slope outputs respond differently to dopaminergic state within the same patient. That pattern would be expected if the two components reflect partly separate mechanisms, as Zarrat Ehsan et al. suggest. This is a descriptive test and not a causal one. The causal perturbation experiments that the imaging literature lacks are outside our scope.

Baselines

The naive reference point is a majority-class predictor together with a single-feature ordinal regressor on mean tap amplitude. That regressor shows how much of the score is recoverable with no temporal structure at all. If our model cannot clearly beat it, the sequence-effect story is moot.

The strongest existing interpretable method is the MediaPipe plus LightGBM pipeline of Zarrat Ehsan et al.. On the personalized Parkinson project finger-tapping dataset (485 videos, 74 patients) it reports 55.95% exact multiclass accuracy and above 94% accuracy within one point, against clinician inter-rater reliability of Krippendorff's alpha 0.62. We rerun it on identical patient-level splits. Its features are recomputed using our cycle boundaries as well as its own, so that segmentation quality does not decide the comparison. Beating it means higher within-one-point accuracy and exact accuracy overall. Given the 0.62 reliability ceiling, however, the more meaningful test is on the stratum of videos with low landmark confidence. There we expect a clear gain, and we require the gain to have a patient-level bootstrap interval that excludes zero.

The strongest learned competitor is a fine-tuned global-attention video model, TimeSformer, which Banerjee et al. found competitive on finger tapping. The file reports no finger-tapping-specific number for it on Lacuna. The closest reported figure is V-JEPA2 on Flip Palm at an area under the curve (AUC) of 85.3% with 80.6% accuracy, and the cross-task sensitivity range is 43.2–57.3%. We fine-tune TimeSformer on whole clips with the same pretraining, loss, GPU-hours, and hyperparameter search budget as our model. We also run the GTSN Global Shift design on a ResNet-50 backbone, without Eulerian Video Magnification, since finger tapping is not a micro-motion task.

Two ablations isolate the key ingredient. The first keeps our architecture but uses fixed-duration tokens instead of cycle-aligned tokens. The second removes the cycle-index embeddings and trains on cycle-shuffled clips, which makes the model order-blind by construction. If the order-blind ablation matches the full model's accuracy, the across-cycle receptive field is not doing the work, and the central claim fails.

Why not just…?

Why not just compute amplitude and interval slopes from hand keypoints, as the handcrafted pipeline already does? This is the most serious threat. A one-dimensional keypoint time series already has a receptive field covering the whole trial, so on clean videos the handcrafted pipeline may be near the reliability ceiling set by the raters' alpha of 0.62. We do not claim to beat it there. Our case rests on the low-landmark-confidence stratum, where Zarrat Ehsan et al. themselves report occlusion failures. We also rest it on producing per-cycle trajectories where keypoints are absent. If stratified results show no gain on hard videos, the honest conclusion is that pose is sufficient, and we would report it.

Why not just fine-tune a video foundation model end to end instead of using a frozen backbone? Fine-tuning may close much of the sensitivity gap reported by Banerjee et al., and it is included as a baseline under matched budget. What fine-tuning alone does not guarantee is that the model uses cycle order rather than mean-amplitude shortcuts. The shuffle test measures exactly this. If fine-tuned TimeSformer is both accurate and order-sensitive, the cycle-token architecture is unnecessary, and the diagnostic becomes the main contribution.

Why not just use a global temporal shift network like GTSN? Global shifting across fixed segments gives temporal reach. Fixed segments, however, straddle cycle boundaries at arbitrary phases, and GTSN relies on optical flow preprocessing. The fixed-token ablation of our own model tests the segment-versus-cycle question directly, and GTSN is run as a full baseline.

Proposed Experiments and Decisive Checks

The primary dataset is the personalized Parkinson project finger-tapping set, evaluated with patient-level five-fold cross-validation. Metrics are exact accuracy, within-one-point accuracy, and ordinal mean absolute error. The 32,847-video screening benchmark is a secondary test for the finger-tapping task if access permits. There we report AUC, sensitivity, and specificity, with sensitivity at matched specificity as the key figure.

Three results decide the proposal. The first is the shuffle test run on every baseline. It succeeds if our model's normalized order sensitivity is substantially above that of fixed-token and whole-clip models, while the order-blind ablation shows near-zero sensitivity. The second is the stratified comparison against the handcrafted pipeline on low-landmark-confidence videos. The third is the within-patient medication-state analysis of amplitude-slope versus interval-slope outputs, reported as a descriptive dissociation with patient-clustered intervals.

A negative result on the first check would itself be informative. If no model, including ours, is order-sensitive, then current severity labels may not reward the sequence effect. That would be a finding about the clinical labels, not about architectures.