Federated Learning of a Soft-Tissue Drift Prior for Wearable Motion Capture

When a motion sensor is strapped to the skin, it does not move with the bone beneath it. Skin slides, fat wobbles, and muscles bulge as the joint bends, so the sensor's orientation relative to the bone changes with posture and effort. Biomechanists call this the soft tissue artifact. It sets an error floor that better sensors cannot remove. The artifact also differs from person to person, because it depends on tissue thickness, muscle mass and body shape. The people whose artifacts matter most, such as children, patients with high body fat, and post-operative rehabilitation patients, are the people whose motion recordings are hardest to pool centrally.

This note proposes to learn, across many private devices, a shared statistical model of how soft tissue makes sensors drift. Each device fits that model locally without any ground-truth motion capture, using only the fact that two sensors on adjacent limb segments must agree about the joint that connects them. Only the parameters of the drift model leave the device. The shared drift model then serves two purposes. It acts as a prior that keeps data-poor users from overfitting when they calibrate. It also acts as a realistic noise generator for retraining the central pose estimator, replacing the hand-designed random drift used today.

Core Claim

Current methods treat soft tissue artifact in one of two ways. Some treat it as generic noise to be simulated centrally. Others treat it as a per-user offset to be calibrated locally. We claim that the population-level structure of the artifact is the missing piece: how offsets depend on joint angle and speed, and how that dependence varies with a few hidden tissue traits. That structure can be learned federatedly from unlabeled wear data. Two measurable consequences follow. First, pose estimators trained with drift sampled from the federated prior should beat estimators trained with randomized drift on held-out real subjects. Second, users with only a few minutes of data should calibrate better with the prior than without it, and no worse than with no personalization at all. The second prediction directly targets the "fairness trap" of personalized federated learning.

What Prior Work Does and Where This Differs

Three centralized lines of work motivate the proposal by showing that the artifact is real, structured, and person-dependent. MODA augments clean motion data with time-varying rotation and translation offsets. On the CIP dataset it improves angular error by 20% and SIP error by 11%. Its authors state, however, that the offsets are randomized and that biomechanically informed drift models remain future work. Our method supplies such a drift model, learned from real wearers instead of hand-specified. Loose Inertial Poser learns a latent autoencoder of clothing-induced secondary motion from paired tight and loose recordings plus cloth simulation. That requires paired central data we will not have, and it models fabric rather than tissue. Shape-aware Inertial Poser shows that inertial signals depend on body shape, cutting joint error on the MID children-and-adults dataset from 8.18 cm to 5.24 cm. It corrects for rigid segment scaling, not for skin sliding, and it assumes central access to sensitive morphology data from children.

The label-free local fit borrows from dynamic calibration. The Transformer IMU Calibrator separates sensor-to-bone offset from global heading drift, using the fact that bone-frame linear acceleration does not depend on the mounting rotation. It also pauses calibration when rotations are not diverse enough. We reuse both ideas as ingredients of the local objective. TIC, though, is trained with supervision and centrally, and it estimates a static offset rather than a posture-dependent one.

On the federated side, FedHealth freezes a shared feature extractor, fine-tunes classification layers per user, and aligns feature covariances. It reports 99.4% activity-recognition accuracy on UCI HAR, a 5.3-point gain over no personalization. It classifies activities from abstract time series and has no notion of sensor placement. Federated Learning with Partial Model Personalization splits a network into shared and personal parts, and finds that personalizing input layers suits sensor-like variation. It also warns that data-scarce clients can end up worse than the shared model. That warning constrains our design. The obvious transfer of this work to motion capture would be a shared skeletal network with a personal input adapter trained on a self-supervised loss. We do something different. The personal parameters are an explicit, physically interpretable offset trajectory and a low-dimensional tissue code. The shared object is a generative model of offsets rather than a network trunk. The personal fit is regularized by that shared prior instead of being left free. Meta-HAR and SemiPFL share the goal of separating shared structure from user style, but they work on activity labels. None of these federated works estimates continuous joint kinematics.

Research Program

We write the rotation offset between sensor and bone on segment $s$ for client $k$ at time $t$ as

$$\Delta R_{k,s}(t) = \exp\big(g_\phi(q_k(t), \dot q_k(t), z_k)\big)\, R^{0}_{k,s},$$

where $q_k(t)$ and $\dot q_k(t)$ are the estimated joint angles and velocities from a frozen pose estimator, and $z_k$ is a low-dimensional tissue code private to client $k$ (4 to 8 dimensions). $R^{0}_{k,s}$ is the static mounting rotation, also private. $g_\phi$ is a small network with shared weights $\phi$ that outputs an axis-angle vector, and $\exp$ maps that vector to a rotation. A translation offset is modelled the same way.

On device, the offsets are fit with no motion capture labels. The fit minimizes a rigid-body consistency residual $\mathcal{L}_{\text{joint}}$. Two sensors on adjacent segments, once de-rotated by their offsets, must report accelerations consistent with a common joint center, given the measured angular velocities and learned lever arms. Where the joint is hinge-like, as at the knee, their angular velocities must agree off the hinge axis. The local problem is

$$\min_{z_k, R^{0}_k, \phi}\; \mathcal{L}_{\text{joint}}(z_k, R^{0}_k, \phi; \mathcal{D}_k) + \lambda \lVert z_k \rVert^2,$$

where $\mathcal{D}_k$ is the client's unlabeled sensor data and $\lambda$ weights a standard Gaussian prior on the tissue code. We follow TIC's diversity rule and use only windows with enough rotational variety.

Training alternates in the manner of FedAlt. Clients first update their private $z_k$ and $R^0_k$ with $\phi$ fixed, then take a few steps on $\phi$, and the server averages $\phi$. The server never sees codes, offsets or signals. After training, the server samples $z \sim \mathcal{N}(0, I)$, generates offset trajectories along motion sequences from AMASS (the large public motion capture archive used by MODA, PIP and LIP), and retrains the pose estimator on sensor signals corrupted by these offsets. At deployment, a new user's device fits only $z_k$ and $R^0_k$, a handful of parameters, which is where the prior should prevent overfitting.

The analysis has three threads. The first tests whether $\mathcal{L}_{\text{joint}}$ can identify posture-dependent offsets at all. This is checked on the bone-pinned cadaver dataset, where the true artifact is essentially zero. The second tests whether the learned $z$ carries tissue information, by probing it against shape and age on MID. The third measures what the shared $\phi$ leaks, by attempting to recover client body shape from single-client updates.

Baselines

The naive reference is a pose estimator trained centrally on synthetic AMASS signals with no drift augmentation and no personalization. It is the PIP-style kinematics network from Physical Inertial Poser, which reports a 15% pose-error reduction over TransPose. We run the same network with its physics layer both on and off. Every other method shares this architecture and training budget, so it isolates what the drift model adds.

The strongest centralized competitor is MODA augmentation on the same network with the same AMASS epochs. The only difference is that its random offset distribution replaces our federated prior. The number to beat is MODA's reported CIP gain of 20% angular error and 11% SIP error over no augmentation. Our method must beat MODA's absolute errors on held-out subjects, not merely match its improvement over the naive model.

On MID we also compare with SAIP, whose reported joint error is 5.24 cm. SAIP targets shape variation, so we additionally run SAIP's retargeting combined with our prior, to test whether the two gains add up.

The strongest local competitor runs a TIC-style static offset calibration on each device independently. The file gives no reported calibration error for TIC on DSTIC ("no reported result found on Lacuna for TIC calibration error"), so we reimplement it with the same unlabeled windows.

The strongest federated competitor is FedAlt partial personalization of the pose network's input layer, trained with the same $\mathcal{L}_{\text{joint}}$ loss. It receives the same number of rounds, local steps, and client data, and its learning rate and $\lambda$ are tuned on the same validation clients.

Two ablations isolate the key ingredient. The first keeps the drift model's form but fits $\phi$ on each client alone with no averaging. This removes federation while keeping the physical structure. The second removes the posture dependence, so that $g_\phi$ outputs a constant offset. This tests whether the dynamic part of the artifact matters.

Our method counts as beating a baseline if it lowers mean SIP and angular error on held-out subjects. It must also lower the worst-decile per-client error relative to no personalization, and do so in the few-minutes-per-client regime.

Why not just…?

Why not just pool anonymized sensor data centrally and train with MODA plus SAIP? This is the most serious threat to the augmentation half of the claim. If random drift is already rich enough, a learned prior adds nothing. The motivation for our approach is MODA's own statement that its perturbations are randomized and not biomechanical, and SAIP's evidence that morphology shifts the signals substantially. The governance argument, that gait traces act as biometrics and child morphology data is sensitive (Opportunities and obstacles for deep learning in biology and medicine), explains why pooling is often impossible. It does not show that pooling would perform worse. We therefore also run a centralized oracle that fits the same drift model on pooled data. If the federated version comes close to this oracle, and both beat MODA, the case is made. If MODA ties the oracle, the learned-prior idea fails regardless of federation.

Why not just calibrate each user locally, as TIC does, and skip federation? For users with an hour of varied motion, local fitting may be enough, and we expect only a small gap there. The claim rests on data-scarce users. A rehabilitation patient may perform a few repetitions of one exercise, which gives little rotational diversity, so an unconstrained local fit is poorly determined. We sweep per-client data from one minute to sixty minutes on DSTIC and MID. If local-only calibration matches ours at every budget, the federated prior is unnecessary and we will report that.

Why not just personalize an input adapter with FedAlt, following Pillutla et al.? That approach is the natural federated default, and its own paper documents that adapters degrade on data-scarce clients. An adapter has thousands of free parameters. Our personal state is a tissue code plus one static mounting rotation per sensor, tied to a shared generative model. The head-to-head comparison at matched loss and budget, broken down by client data size, settles which parameterization avoids the fairness trap.

Why not just use joint and chain constraints with a Kalman filter, as in the postoperative orthopedic pipeline? That pipeline (Postoperative human motion evaluation) reports a 1.23° wrist range-of-motion error. Its authors nonetheless say soft tissue interference remains unsolved, and the system relies on GPS drift correction that is unavailable indoors. Our local objective uses the same kind of constraint, so we include that pipeline as a non-learned reference on the knee-sleeve data.

Proposed Experiments

We simulate federation by treating each subject as a client, which produces naturally non-identical data. On MID this gives 20 clients split between children and adults. DSTIC gives 5 clients with long recordings each. The Intelligent Knee Sleeves dataset gives clients defined by subject and sleeve size; its stacked recurrent model reports 7.21° average joint error. CIP allows direct comparison with MODA. Motion capture ground truth is withheld from clients and used only for scoring.

The metrics are SIP error, angular error, joint position error and jitter. Each is reported as a mean and as the worst decile across clients, along with communication cost per round. One caveat applies throughout: optical markers on living subjects are themselves on skin. Errors on MID, DSTIC and CIP are therefore agreement with a reference that also carries artifact.

For that reason the first decisive check uses the bone-pinned cadaver dataset. There the sensors are screwed into bone, and the femur sensor-to-bone misalignment is 0.98° ± 0.55°. Across its 53 flexion trials, our local fit should recover a nearly constant offset whose posture-dependent part is well under a degree. If it invents large posture-dependent drift on rigid mounts, the consistency loss is absorbing errors from the pose estimator, and the method is not measuring tissue at all.

The second check is the data-budget sweep: our method must beat both local-only calibration and FedAlt in worst-decile error at low budgets.

The third check is leave-children-out. We train $\phi$ on adult clients only, then test few-minute calibration on children. This tests whether the prior's tissue code extrapolates, or whether SAIP-style shape handling is still needed.

The fourth is the augmentation comparison. We retrain the pose network with drift sampled from the federated prior, from MODA's distribution, and from the centralized oracle, and evaluate all three on CIP and held-out MID subjects.

The last check probes privacy. An attacker is given single-client $\phi$ updates and tries to predict adult versus child. If it succeeds well above chance, we add clipped, noised aggregation and report the resulting accuracy cost.