Kinematic Deception: Hijacking the Attention of Pose-Based Intention Predictors

When an autonomous car or an assistive robot watches a person, the most valuable signal about what that person will do next is not where they are, but how they move. A torso rotation, a weight shift onto the forward foot, or the height of the wrist partway through a reach all reveal the person's goal several hundred milliseconds before their body's center actually moves — this is the core finding of kinematic intention encoding, and modern prediction models exploit it by feeding sequences of skeletal joint positions ("pose") into transformers, whose internal attention mechanism learns which joints and which moments carry the most predictive information (Social-Pose: Enhancing Trajectory Prediction with Human Body Pose). Separately, adversarial machine learning has shown that transformer attention can be deliberately hijacked: an adversary who inflates the attention assigned to one irrelevant input region can blind the model to everything else, collapsing its predictions (Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?).

These two literatures have never met, and their meeting exposes an attack surface that is behavioral rather than pixel-level. A pedestrian, a robot coworker, or any observed human can choose their own motor plan — no imperceptible noise injection required — to redirect a predictor's attention away from the joints that reveal their true intention. I propose to build and study exactly this attack, which I call kinematic attention hijacking: optimizing joint-angle trajectories through a differentiable model of the human skeleton so that the resulting motion is biomechanically legal, looks natural, yet forces a pose-based predictor's spatiotemporal attention onto uninformative joints (or onto fake preparatory cues), producing wrong intention and trajectory forecasts. The proposal includes the matching defense — attention-preservation training adapted to skeletal sequences — and a mechanism study that tests whether the "degeneration" phenomenon observed in robust trajectory prediction is, in fact, unnoticed attention redirection.

Core Claim

The claim has three parts. First, transformer and graph-attention models that consume human pose are structurally vulnerable to attention redirection through the actor's own legitimate movement: because intention is encoded in smooth, low-frequency kinematic modulation — wrist height at 70% of movement duration, hand-plane angle (Decoding intentions from movement kinematics) — a deceptive actor can work in exactly the same low-frequency band, so no filtering rule can remove the attack without also removing genuine intention signals. Second, this attack is cheap: it needs no adversarial patch, no pixel access, and no perturbation budget in the classical sense, only control over one's own body, which every real adversary already has. Third, defenses must therefore operate at the attention level, not the input level: extending attention rectification and preservation (Attention, Please! Adversarial Defense via Activation Rectification and Preservation) from static images to spatiotemporal joint sequences should recover robustness where smoothing and standard adversarial training do not.

Research Program

The attack is an optimization over joint angles, not coordinates. Let $\theta_0$ be the attacker's recorded joint-angle trajectory (rigid bone lengths fixed by forward kinematics, so bone-length violations are impossible by construction), and let $A(\theta)$ denote the victim model's internal attention maps induced by the skeletal sequence that $\theta$ produces. The attack solves

$$ \min_{\theta}\; \lambda_{\text{task}}\, L_{\text{task}}\big(f(\theta)\big) + \lambda_{\text{attn}}\, D\big(A(\theta_0)\,\|\,A(\theta)\big) + \lambda_{\text{jerk}}\, J(\theta) $$

where $f$ is the victim predictor, $L_{\text{task}}$ is either an intention-flip loss or a trajectory-error inflation loss, $D$ is a divergence between clean and perturbed attention (following the attention-divergence objective of Attacking Attention of Foundation Models Disrupts Downstream Tasks and the dual attack in Attention, Please! Adversarial Defense via Activation Rectification and Preservation), $J$ penalizes jerk so the motion stays smooth, and the search is constrained to joint-angle limits. The constraint set is what makes this different from all prior work: Adversarial Interaction Attack: Fooling AI to Misinterpret Human Intentions perturbs skeleton coordinates directly on recurrent and temporal-convolutional models with no bone-length or attention modeling (its authors list the missing physical constraints as a limitation), while AdvDO: Realistic Adversarial Attacks for Trajectory Prediction enforces physical feasibility but only for rigid 2D vehicle bicycle dynamics and only through task-level error — it never touches attention. This proposal is, to my knowledge, the first to put articulated human biomechanics and attention redirection in the same optimization.

I will study two attack modes that come directly from the intention-encoding literature. Blinding mode drives attention away from the intent-revealing extremities that pose transformers prioritize — Social-Pose showed attention mass concentrates on ankles, knees, wrists, and elbows as leading indicators of turns — toward a decoy joint, the kinematic analog of Patch-Fool's single-patch hijack. Spoofing mode is subtler: following the finding that the upcoming goal modulates early-phase kinematics (The visible face of intention: why kinematics matters) and that a sparse set of geometric tells (wrist height, hand plane) suffices for decoding (Decoding intentions from movement kinematics), the attacker deliberately performs the preparatory kinematics of a false goal — e.g., a pedestrian who intends to cross shapes their gait to match the "waiting" signature — while still being able to execute the true plan. Spoofing exploits the very encoding the field has spent a decade validating.

Three analyses round out the program. First, a mechanism study: Robust Trajectory Prediction against Adversarial Attacks hypothesizes that attacked prediction models "degenerate" and stop listening to input history, but observed this only through output distributions; I will instrument the attention paid to history and social-context tokens during attacks and test whether degeneration is attention redirection wearing a different name. Second, an uncertainty audit: TrEP: Transformer-Based Evidential Prediction for Pedestrian Intention with Uncertainty assumes ambiguous inputs produce a spike in evidential uncertainty, while Attention-Imperceptible Backdoor Attacks on Vision Transformers shows attacks can hide inside naturally high-attention zones without leaving a residual — I will test whether the hijacking attack can flip the predicted intention while keeping TrEP-style uncertainty low, a regression analog of the ghost-certificate failure (Certified but Fooled! Breaking Certified Defenses with Ghost Certificates). Third, the defense: an attention-preservation loss that penalizes divergence between clean and adversarial spatiotemporal attention maps, combined with kinematic augmentation in the training data, in the spirit of RobustTraj's regularization but operating on attention rather than latent distributions.

Proposed Experiments

The victim models will be reimplemented faithfully: a Social-Pose-style transformer pose encoder feeding a trajectory backbone, evaluated on the JTA benchmark (a synthetic urban pedestrian dataset with 10 million pose annotations), where the original paper reports a 25% Average Displacement Error improvement and 29% Final Displacement Error improvement over point-mass baselines at roughly 2% computational overhead (Social-Pose: Enhancing Trajectory Prediction with Human Body Pose); a TrEP-style evidential intention transformer on PIE and JAAD, where TrEP reports 0.92 intention-prediction accuracy on PIE and a 9% AUC gain on JAAD (TrEP: Transformer-Based Evidential Prediction for Pedestrian Intention with Uncertainty); and a TRiPOD-style graph attention network on 3DPW and PoseTrack, where TRiPOD reports 150.41 cm error at a 900 ms horizon versus 208.31 cm for decoupled baselines (TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild). Primary metrics: intention flip rate and ADE/FDE inflation as a function of joint-angle deviation budget (in degrees per joint), attention-map divergence, cross-architecture transfer rate (against which AdvDO's reported 77% transfer between trajectory predictors is the reference point), and evidential-uncertainty response. Naturalness will be checked with a human perceptual study modeled on the occlusion paradigm of Decoding intentions from movement kinematics, whose human observers achieved only 0.608 AUC (0.677 on high-transparency movements) at distinguishing intentions — the bar a deceptive trajectory must stay under is low, which is precisely the danger.

Baselines

The naive reference point is clean-data performance with no attack: the reimplemented Social-Pose and TrEP models at their reported operating points (25% ADE improvement over point-mass baselines on JTA for the former; 0.92 intention accuracy on PIE for the latter). Any attack that materially degrades these numbers while remaining physically feasible demonstrates the vulnerability; the do-nothing row also calibrates how much clean accuracy the defense is allowed to spend, for which RobustTraj's reported 3% clean degradation is the tolerance standard (Robust Trajectory Prediction against Adversarial Attacks).

The strongest existing attack competitor is an AIA-style attack: projected-gradient-descent perturbation of joint sequences with a temporal-smoothness penalty, which achieved 100% white-box attack success on temporal-convolutional and recurrent interaction models with perturbations of only 2% to 5% of the total depth range, and over 80% transfer from one architecture to another (Adversarial Interaction Attack: Fooling AI to Misinterpret Human Intentions). I will port this attack faithfully to the transformer victims, give it the same perturbation budget (measured as joint deviation relative to limb length), the same optimization steps and compute, and the same access to the victim's gradients. My attack beats it if, at equal or smaller kinematic deviation, it achieves a higher intention-flip rate and better transfer on attention-based architectures, and if its motions are rated more natural by human observers — the AIA attack's unconstrained coordinates produce the joint jitter its own authors flag as a weakness.

The critical ablation isolates the attention term: the same biomechanically constrained optimizer with the identical task loss but the attention-divergence term removed — an "AdvDO-for-humans" that maximizes only prediction error through feasible motion, since AdvDO shows exactly this recipe inflates trajectory-prediction error by more than 50% for vehicles (AdvDO: Realistic Adversarial Attacks for Trajectory Prediction). If the attention term adds nothing over this ablation, the proposal's central mechanism is wrong and I will say so. On the defense side, the strongest competitor is RobustTraj's adversarial training with latent regularization, which improved adversarial-data performance by 46% at a 3% clean cost and beat trajectory smoothing by 21% under attack (Robust Trajectory Prediction against Adversarial Attacks); my attention-preservation defense must beat it under kinematic hijacking at no worse clean cost, under the same training budget.

Why not just…?

Why not just run standard $\ell_p$ coordinate attacks instead of modeling attention? Three cited results say this falls short in this setting. Unconstrained coordinate perturbations produce physically impossible or visibly jittery motion that plausibility checks and humans reject, which is why AdvDO moved to a differentiable physical model and why AIA had to add an explicit smoothness loss (AdvDO: Realistic Adversarial Attacks for Trajectory Prediction; Adversarial Interaction Attack: Fooling AI to Misinterpret Human Intentions). And diffuse noise is precisely what transformers tolerate well: Patch-Fool showed Vision Transformers resist random patch noise while collapsing — DeiT-S robust accuracy fell to 6.25% versus ResNet-50's 24.00% — only when the attack was made attention-aware (Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?). Still, I will run the $\ell_p$ attack as a baseline rather than argue around it, and the constrained-ablation comparison above settles the attention question empirically.

Why not just smooth or filter the input trajectories? This is the alternative the attack is explicitly designed to defeat. RobustTraj already showed heuristic trajectory smoothing loses 21% to robust training under attack and costs 9% on clean data (Robust Trajectory Prediction against Adversarial Attacks). More fundamentally, the intention signal itself lives in smooth, low-frequency geometric parameters like wrist height at 70% of movement duration (Decoding intentions from movement kinematics), so my attack operates in the same band; a decisive experiment applies progressively stronger low-pass filtering and shows the attack's flip rate and the model's clean accuracy fall together, with no filter setting that removes one and keeps the other.

Why not just use standard adversarial training? Because the cited evidence says the routing vulnerability survives it. Patch-Fool reports that adversarially trained Vision Transformers remain more fragile than CNN baselines under patch attacks since the underlying attention-routing weakness is untouched (Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?), and RobustTraj found vanilla adversarial training on stochastic predictors fails through gradient obfuscation, requiring an architecture-specific regularization (Robust Trajectory Prediction against Adversarial Attacks). Attention-preservation training with kinematic augmentation will nonetheless be compared head-to-head against standard adversarial training at equal budget, so if plain training suffices against these attacks, the experiments will reveal it.

Why not just drop pose and use point-mass $(x,y)$ trajectories, removing the attack surface? This is the one alternative I take seriously as a real threat. It works — point-mass models have no joint attention to hijack — but it forfeits exactly the predictive power that motivated pose in the first place: Social-Pose attributes a 25–29% error reduction and a 37% collision reduction in social navigation to articulated pose (Social-Pose: Enhancing Trajectory Prediction with Human Body Pose), and TRiPOD's joint forecasting cuts 900 ms-horizon error from 208.31 cm to 150.41 cm (TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild). The experiments will settle the trade-off directly: if a defended pose model under attack still beats an undefended point-mass model, pose is worth keeping; if not, that is an honest and important negative result for the field.

Decisive Checks

The proposal stands or falls on four measurements. Whether attention divergence predicts misprediction: across attacks and victims, the correlation between clean-vs-attacked attention divergence and forecast error should be strong if redirection is the mechanism, weak if it is incidental. Whether the attention term beats the equally-constrained task-only ablation at matched budgets on the JTA, PIE, and 3DPW setups. Whether any filter setting separates attack removal from signal removal — the low-frequency argument says no. And whether attention-preservation training restores performance under hijacking at no more than RobustTraj's 3% clean cost, while leaving the evidential layer of a TrEP-style model either genuinely flagging the attacks or exposing a ghost-certificate-style blind spot worth reporting in its own right. A secondary but scientifically valuable outcome is the mechanism result: if degeneration in conditional predictors (Robust Trajectory Prediction against Adversarial Attacks) turns out to be attention redirected away from history and social tokens, two literatures that currently do not cite each other acquire a shared explanation, and the defense design space for both narrows usefully.