PRISM: Profiling and Representation of Individual Stereotypical Motion

Austin A. MBaye, Savik Kinger, Matthew S. GoodwinarXiv 2026 · arXiv

In the field of computer vision for behavioral analysis, we are often satisfied with simply "labeling" an action. If a system correctly identifies a clip as "body rocking," we call it a success. But for clinicians working with autistic individuals, a label is just the beginning. Two instances of the same behavior can differ wildly in their intensity, speed, and regularity—nuances that are critical for diagnosis and monitoring treatment progress.

The paper "PRISM: Profiling and Representation of Individual Stereotypical Motion" introduces a framework to move beyond categorical labels, providing a high-fidelity "profile" of how repetitive movements are actually expressed.

TL;DR

PRISM is a new framework that uses dense optical flow and tracked masks to quantify the "how" of repetitive movements through four key metrics: intensity, spatial extent, frequency, and rhythmicity. By moving beyond simple categorical labels, it reveals significant behavioral variability within individuals that standard pose-estimation methods often miss.

What Problem Does This Paper Address?

Current automated behavior analysis typically relies on two things: categorical labels (e.g., "hand flapping") and anatomical pose estimation (skeletons). The authors argue that both are insufficient for clinical use.

First, labels hide variability; "body rocking" might be a slow, rhythmic sway one day and a violent, erratic motion the next. Second, skeletal pose estimation is "leaky." It omits body-shape information and fine-grained motion (like finger movements) that aren't captured by a fixed set of 17 joints. Furthermore, pose estimators are prone to tracking errors and are highly sensitive to processing choices like smoothing and confidence thresholds.

PRISM aims to provide a human-centric, regional representation that is robust to these issues while remaining interpretable for clinicians.

Methods

The PRISM (Profiling and Representation of Individual Stereotypical Motion) framework processes video bouts using a multi-stage pipeline:

  1. Tracked-Region Motion: Instead of joints, PRISM uses tracked person masks and camera-corrected dense optical flow.
  2. The Four Pillars of Profiling:
    • Intensity ($I_b$): Measures torso-relative speed using the spatial upper quartile of flow vectors.
  3. Spatial Extent ($E_b$): Quantifies how much the person's silhouette varies in space (RMS orbit radius).
  4. Frequency ($\hat{f}_b$): Estimates the repetition rate using signed-flow recurrence.
  5. Rhythmicity ($PS_b$): This is the most novel metric. It uses persistent homology (specifically Vietoris-Rips persistence) on sliding-window embeddings of the flow trajectory to score how "regular" or "cyclic" the movement is.
  6. Interpretability via Decoding: To make these numbers concrete, the authors trained a "profile-conditioned decoder" that can take a numerical PRISM profile and generate a corresponding skeletal animation, allowing clinicians to "see" what a specific set of metrics looks like.
Figure 1. PRISM overview. The tracked mask defines target-person support. Torso-relative flow yields intensity; camera-corrected total signed flow yields recurrence frequency and sliding-window periodicity. Common-grid masks independently yield frame-wise RMS extent. Event boundaries yield duration and between-bout timing.

Datasets

The authors evaluated PRISM across a diverse range of data:

  • Synthetic Benchmark: 1,890 video test cases generated with "Skeleton Lab," featuring controlled variations in speed, amplitude, and frequency for behaviors like arm flapping and head banging.
  • Real-World Clinical Data: 1,404 expert-annotated bouts of stereotypical motor movements (SMMs) from six autistic participants, recorded in laboratory and classroom settings.
  • External Audit Data: 36,930 clips from the ASDMotion dataset used to benchmark the limitations of standard pose estimation.

Key Insights and Ideas

  • The "Complementarity" Proof: The authors mathematically demonstrate that in a simple oscillator model, you need at least three factors (intensity, extent, and frequency) to uniquely identify the physical parameters of the movement. Removing any one creates ambiguity.
  • Pose is Not Enough: The paper highlights that even "perfect" COCO-17 trajectories can fail to distinguish between visibly different movements if the motion happens between the landmarks (e.g., hand shape changes).
  • Rhythmicity vs. Frequency: A movement can have a clear frequency but be highly irregular. By using persistent homology, PRISM can distinguish a steady, rhythmic rock from a "stuttering" or interrupted one, even if their average frequencies are similar.
  • Privacy-Preserving Analysis: Because PRISM relies on masks and flow rather than raw RGB pixels for its core metrics, it offers a path toward sharing behavioral profiles for collaborative research without necessarily distributing sensitive participant footage.

Key Results

  • High Fidelity: On synthetic data, PRISM recovered generating parameters with high accuracy (Spearman $\rho \approx 0.95$ for intensity and extent).
  • VLM Comparison: PRISM significantly outperformed state-of-the-art Vision-Language Models (like Grok and Qwen-VL) in frequency estimation. PRISM achieved an 85% success rate (within 20% of ground truth) at 10 fps, while the best VLM reached only 47.5%.
  • Revealing Hidden Variability: In real-world data, PRISM successfully identified bouts that shared the same label but differed significantly in expression. For example, two "body rocking" bouts for the same participant might have identical intensity but differ by 40% in frequency and have completely different rhythmicity scores.
Figure 2. Complementary real-video measurements. Selected bouts share participant, behavior, and source recording. The shaded coordinate is matched within six percentile points, while other coordinate ranks can differ. Matching one measurement does not establish equivalent underlying movement or exclude observation error.

Limitations and Future Work

  • Detection vs. Profiling: PRISM is a profiling tool, not a detection tool. It assumes that the start and end times of a behavior bout have already been identified (either by a human or another model).
  • Tracking Dependency: The system's accuracy is tied to the quality of the initial person tracking and masking. While robust, extreme occlusions or very poor lighting could degrade the flow signal.
  • Clinical Validation: While the technical metrics are sound, the authors note that the "clinical significance" of these variations (e.g., does a change in rhythmicity predict a change in anxiety?) remains a subject for future longitudinal studies.
  • Generalization: Future work could expand the framework to more diverse "non-stereotypical" movements or different clinical populations.

Summary updated

Abstract

An anatomical pose provides interpretable joint structure, but no fixed set of physical landmarks can fully specify a person's movement. We introduce PRISM, a complementary representation that characterizes repetitive movement using tracked masks and dense optical flow. Torso-relative flow measures intensity, frame-wise mask variation measures spatial extent, signed-flow recurrence estimates frequency, and persistent sliding-window geometry describes the rhythmicity of the movement. Across 1,890 synthetic conditions, the measurements recover controlled speed, amplitude, and frequency while exposing observation limits. An idealized hand-motion experiment demonstrates variation omitted by identical exact COCO-17 trajectories; controlled corruption experiments separately examine measurement sensitivity. Audits of two pose methods on the same data cohort and 36,930 external skeleton clips document incomplete observation and dependence on processing choices. In 1,404 annotated complex stereotypical motor movement bouts from six autistic participants, total flow converges with pose motion, while matched-coordinate examples and repeated-session profiles show differences in intensity, extent, frequency, and rhythmicity within behavior labels. A separate VLM comparison evaluates quantitative frequency estimation, and a profile-conditioned decoder links the measurements to illustrative skeletal movements. PRISM connects numerical profiles to visible examples, providing an interpretable way to compare how the same behavior is expressed beyond its label or event count.
BibTeX
@misc{mbaye2026prism,
  author = {Austin A. MBaye and Savik Kinger and Matthew S. Goodwin},
  title = {{PRISM: Profiling and Representation of Individual Stereotypical Motion}},
  year = {2026},
  eprint = {2611.23646},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23646},
}
Figure