Part-and-Phase Concept Bottlenecks for Locating Movement Errors from Whole-Sequence Labels

Physiotherapists, coaches and judges rarely label a movement frame by frame. They say "this exercise was wrong, error type 2" or "this routine scored 8.4," and the learner is left to guess which limb and which moment went wrong. This is the same problem that econometricians call aggregation bias: when fine-grained behavior is observed only through a coarse summary, the local variation that would point to a fix disappears. Skeleton-based models, which treat the body as a graph of joints linked by bones and evolving over time, are good at predicting these coarse labels. Their explanations, however, are attention maps or saliency scores read off after the fact, and nobody can edit those to correct a prediction.

We propose a model in which the whole-sequence prediction is, by construction, a sum of contributions from individual body parts during individual movement phases. Each contribution comes from a small set of named kinematic quantities: elbow flexion, trunk rotation velocity, left–right symmetry. The model is trained only on sequence-level labels. Our claim is that this additive part-and-phase bottleneck recovers where and when an error happened, matches black-box accuracy, and lets an expert overwrite one part's concept values at one phase and see the prediction change. The direction of interest is localization, not raw accuracy. We test localization directly with injected errors whose true location is known.

Core Claim

The spatio-temporal graph network is the encoder. The skeleton graph sets the spatial units. The granularity limitation is the training regime: labels are coarse, and the quantities we want are fine. Interpretability is the constraint on the readout. Concretely, a spatial-temporal graph convolutional network in the style of ST-GCN encodes the skeleton. Its features are pooled into $P$ body parts, using the same arm, leg and spine groupings as the hypergraph of the Low Back Pain Transformer, and into $T$ soft temporal phases. For each part $p$ and phase $t$, a small head predicts a low-dimensional, quantized vector $c_{p,t}$ of named kinematic concepts. The class score is then forced to pass only through these vectors:

$$ s_k = b_k + \sum_{p=1}^{P}\sum_{t=1}^{T} a_{p,t}\, g_{k,p}\!\left(c_{p,t}\right) $$

The symbols are as follows. $s_k$ is the score for class $k$ and $b_k$ is a bias. $a_{p,t}\ge 0$ is a phase-importance weight that sums to one over $t$ for each part. $g_{k,p}$ is a small shape function in the style of a generalized additive model, so each concept's effect can be plotted as a curve. The product $a_{p,t}\,g_{k,p}(c_{p,t})$ is the explanation itself: a part-by-phase map whose entries add up exactly to the prediction. When the concepts are inputs, the map involves no post-hoc approximation. The open question, which our experiments test, is whether the encoder routes extra information through the concept values.

Kinematic concepts are given auxiliary supervision. The targets are analytic values (joint angles, angular velocities, contralateral differences) computed from smoothed skeletons. Because the encoder sees raw, possibly noisy skeletons, it must predict concepts robustly rather than compute them trivially. Phase weights use the high-variance Gaussian query initialization and the attention-divergence loss of Interpretable Long-term AQA. That paper shows that without such a loss, long-sequence attention collapses into a uniform mean, with Spearman correlation falling from 0.842 to 0.628 on Rhythmic Gymnastics. Its temporal-only "weight × score" head is the closest existing readout. We add the spatial (body-part) axis and the named-concept axis.

Research Program

The work has three parts.

The first part is the architecture and its leakage controls. Hi-CBM shows that soft concept probabilities can covertly encode the class and bypass the bottleneck. We therefore quantize $c_{p,t}$ to a few levels per concept and remove every residual path from encoder to classifier. We also measure leakage directly. The test replaces predicted concepts with their analytic values at test time; if accuracy drops sharply even when the analytic values are accurate, the model has been relying on information beyond the named quantities.

The second part settles the granularity of the units. Fang et al. report that pushing spatial units too fine makes delineation unstable as data becomes sparse. We expect the same at single-joint, single-frame resolution. We therefore sweep $P$ from 2 to 10 parts and $T$ from 2 to 16 phases, and choose the resolution by localization on a held-out set that contains injected errors, not by accuracy.

The third part is test-time intervention. The protocol, borrowed from Concept Bottleneck Models, sets a mispredicted $c_{p,t}$ to its analytic value and measures how often the prediction is corrected.

This proposal differs from PCBEAR in three ways. PCBEAR clusters pose sequences with FINCH into unnamed static and dynamic concepts on a VideoMAE backbone. Our concepts are named biomechanical quantities, our encoder is graph-native, and our readout is localized in space and time. It differs from STExplainer, which learns a minimal predictive subgraph for traffic sensors but exposes no intervenable variables. It also differs from Graph CBMs and Relational CBMs, which model concept dependencies only on static data.

Baselines

The first reference point is a naive one: predicting the majority error class, together with a uniform part-and-phase map for localization. Any localization score must clearly exceed the uniform map.

The strongest accuracy competitor on clinical error classification is the Low Back Pain Transformer. It reports Keraal accuracies of 73.17% (torso rotation), 64.44% (flank stretch) and 74.28% (hiding face) when trained on simulated errors and tested on unseen data. It is also the strongest localization competitor because it already claims joint importance through its attention maps. We will rerun it on identical splits with the same hyperparameter search budget. Beating it means matching its accuracy within one point and exceeding its attention-map localization.

On NTU RGB+D 60, the strongest graph model in the file is DualHead-Net, with 92.0% cross-subject accuracy from four streams; ST-GCN reports 81.5%. We will compare using the same single-stream ST-GCN encoder with and without our bottleneck, so that only the readout differs. An accuracy cost of three points or less against the unconstrained version of the same encoder counts as success. We do not expect to beat a four-stream ensemble. On Penn Action, PCBEAR reports 97.90% against 95.31% for VideoMAE. We will run PCBEAR-style FINCH concepts on our encoder to isolate the effect of named concepts against clustered ones.

The key ablation removes the part axis, giving the temporal-only weight–score head of Interpretable Long-term AQA on the same encoder. A second ablation removes the concept axis, so that $c_{p,t}$ becomes free latent vectors. These two ablations separate what spatial decomposition contributes from what named kinematics contribute.

Why not just…?

Why not just compute joint angles and velocities by hand and fit a sparse linear model or generalized additive model? This is the most serious threat. On clean Kinect 3D skeletons it may match us, and then the deep encoder buys nothing. We run it on every benchmark, giving it the same concepts, parts and fixed equal-length phases. We expect our model to pull ahead in two situations. One is noisy 2D pose, where analytic angles are corrupted; FineGym documents that pose estimators fail on fast motion. The other is phases that are not equal in length across subjects. To test both, we evaluate under controlled injected pose noise. If the hand-built additive model ties us at all noise levels, we will report that the contribution reduces to the evaluation protocol, not the model.

Why not just train a black-box ST-GCN and read localization from gradients or attention? STExplainer reports that post-hoc graph explainers are less faithful than intrinsic bottlenecks on spatio-temporal data, and PCBEAR reports that Grad-CAM fixes on single frames. Neither result is on skeleton error localization, so we test it directly. Gradient and attention maps from the unconstrained encoder are scored against the same injected ground truth as our maps.

Why not just use the temporal weight–score head from long-term action quality assessment? That head localizes when an error occurs but not where. The no-part-axis ablation measures exactly what the spatial axis adds on errors injected into a single limb.

Proposed Experiments

The decisive experiment is a controlled disaggregation test. Starting from NTU RGB+D 60 and Keraal "correct" sequences, we inject localized deviations into a known body part during a known time window, as a new "error" class. The deviations include restricted elbow range, delayed trunk rotation and asymmetric knee flexion. We train with sequence labels only and score the part-by-phase map against the injection mask, using the area under the localization curve and pointing-game hit rate. We repeat the test at three levels of added pose noise.

Faithfulness is measured with deletion curves: we zero the top-ranked part-phase cells and track the drop in the prediction. Leakage is measured with the analytic-concept substitution test. Intervention is measured as the correction rate when the $k$ worst-predicted cells are replaced with analytic values. On real Keraal errors, we ask a physiotherapist to tag which body part each error type concerns, and check whether the dominant part in our map agrees.

The proposal fails if the hand-built additive model localizes as well under noise. It also fails if post-hoc maps from the black-box encoder match ours, or if the bottleneck costs more than three points of accuracy on NTU-60 cross-subject against the identical unconstrained encoder.