Answer at the Granularity the Motion Signal Supports

Many sensing problems ask a model to tell apart classes that produce the same physical trace. A phone's accelerometer feels nearly the same thing on an overground train and on a subway. A car stuck in traffic moves like a bus. A Parkinson's patient at one severity stage walks much like a patient one stage over. Flat classifiers handle this by guessing, and published accuracies average those guesses in with easy cases. Clinical studies handle it by dropping the hard intermediate class altogether. We propose a third option. First, find from data where the motion signal is ambiguous. Then bring in side information, such as location or satellite context for trips and patient attributes for gait, only at those points. Finally, let the model answer at a coarser level ("rail", "moderate-or-worse") whenever even the combined evidence cannot support a fine label.

In two sentences: a signal-only model's own confusion matrix tells us which classes share a motion profile, and we turn it into a coarse-to-fine label tree. Context is added only at the ambiguous branches of that tree, and the model abstains to the parent node when the branch stays uncertain. The result should be compared with standard fusion on how often it is right at each level of detail, not only on flat accuracy.

Core Claim

Fusing context everywhere is the default. Its known cost is that a weakly aligned extra modality can inject noise into classes that did not need it. In the TFormer ablations, adding clinical-image features to the final classifier slightly hurt skin-lesion diagnosis (TFormer). We claim that confusion-routed fusion behaves differently from global fusion in three ways:

  1. It matches global fusion on the confused pairs, such as train/subway and car/bus.
  2. It does not degrade the classes the motion signal already separates.
  3. It degrades more gracefully when context is missing or labels are scarce.

We further claim that backing off to a parent node gives a better trade-off between risk and coverage than either a flat classifier or flat abstention.

We are candid that, on the confused pairs alone, simple global concatenation of a strong context feature may do just as well. That outcome would mean routing contributes robustness rather than accuracy, and the experiments are designed to tell those two cases apart.

Grounding in Prior Work

Evidence that the residual errors are physical, not a lack of model capacity. Several papers locate the failure precisely:

  • Tang and Cheng pair a multi-scale feature pyramid with bidirectional LSTMs on smartphone inertial data. They reach 95.1% accuracy on the Sussex-Huawei Locomotion (SHL) dataset, yet still report train–subway confusion (Feature Pyramid biLSTM).
  • Zhao et al. attribute car/bus/train errors to shared vibration signatures (Short-Term Spatial/Frequency + IndRNN).
  • Zheng et al. and Dabiri et al. show that velocity overlap defeats uniform trajectory slicing on GPS data (Zheng et al., Dabiri et al.).

These papers motivate our premise that a bigger signal model will not close the gap.

The clinical analogue.

  • Xue et al. combine deep ECG features with tabular risk factors by concatenation, reaching AUC 0.847. They exclude 50–89% stenosis because its traces overlap with normal ones (Xue et al.).
  • Ragnarsdóttir et al. see held-out F1 for pulmonary hypertension fall from 0.78 (binary) to 0.63 (three-tier severity) (Ragnarsdóttir et al.).
  • Rohaut et al. find that adding modalities reduces "gray zone" uncertainty in neuroprognosis (Rohaut et al.).

We take a position between Xue and Rohaut. We keep the intermediate class and use context to resolve it where possible. Where context cannot resolve it, we report the coarser answer instead of dropping the class.

Papers that supply the method.

  • DeepTag regularizes a clinical-note classifier with a hierarchical loss over the SNOMED-CT ontology and defers the 10% most ambiguous cases to experts (DeepTag). We reuse the hierarchical loss and deferral idea, with two changes. Our tree is learned from signal confusions rather than taken from an ontology. Our deferral goes to a parent label rather than to a human.
  • TabMixer conditions a video backbone on tabular data through spatial, temporal and channel mixing. It is reported to beat late concatenation, FiLM (a per-channel scale-and-shift conditioning layer) and DAFT (TabMixer). We use it as one candidate conditioning operator, adapted to one-dimensional signals.

TabMixer and TFormer both fuse globally and are evaluated only on imaging. Our contribution concerns where fusion happens, which is a separate question from how it happens.

The closest kinematic-staging work. These parallel 1D-CNN and CNN-transformer gait models are the strongest existing results on the gait benchmark, and neither uses non-kinematic context:

Research Program

Step 1: train the signal-only model. We train a signal-only model $f$ (motion trace to class) with out-of-fold predictions over training subjects. The training folds alone give a confusion matrix $C$, so the test set never shapes the tree.

Step 2: build the label tree. We symmetrize the confusion matrix into $S = C + C^\top$ and cluster classes agglomeratively on $S$. The result is a tree whose internal nodes group mutually confused classes, for example rail = {train, subway} and road-motorized = {car, bus}. Nodes whose within-group confusion exceeds a threshold $\tau$ are marked ambiguous. The threshold $\tau$ is chosen on validation data.

Step 3: route context to ambiguous nodes. The model shares one motion encoder $h(x)$. Each node $v$ has its own head that picks among the node's children. At unambiguous nodes the head sees only $h(x)$. At ambiguous nodes the head is conditioned on a context embedding $g(z)$ through an operator $\phi$:

$$p(c \mid x, z, v) = \mathrm{softmax}_c\big(W_v\,\phi_v(h(x), g(z))\big),$$

Here $c$ ranges over the children of $v$, $z$ is the context vector, and $\phi_v$ is one of three operators: concatenation, FiLM-style scale-and-shift, or TabMixer-style mixing. Missing context is encoded with a learned mask token rather than imputed, following the informative-missingness designs surveyed in Next-gen health.

Step 4: decode with backoff. Training uses the product of node probabilities along the true path plus a DeepTag-style hierarchical penalty. At test time the decoder descends the tree and stops at the deepest node whose path probability exceeds a confidence level $\alpha$. Sweeping $\alpha$ traces a hierarchical risk–coverage curve: error rate against the fraction of samples labeled at leaf level.

Instantiations. We instantiate this on two benchmarks.

On SHL, the motion signal is the inertial channels used by Tang and Cheng. The context is low-rate non-inertial phone data such as location/satellite-availability and cellular summaries. The file describes SHL only as "multi-modal smartphone sensor recordings", so the first task is an audit of which context channels the release actually contains. The cellular-only results of MonoSense and the tower-overlap caveat of Graells-Garrido et al. tell us cellular context alone will be partial.

On the PhysioNet Parkinson's gait data, the motion signal is the 18 force channels. The context is whatever subject-level attributes ship with the records. If the audit finds none beyond the labels, this arm tests only the hierarchy-plus-backoff half of the claim, and we will say so.

For both datasets we use subject-wise splits. With 93 patients, splitting walks from the same person across train and test would inflate results. Our numbers may therefore not be directly comparable to published ones, and we will rerun the published architectures under our splits.

Baselines

Naive reference. The do-nothing reference is the signal-only flat classifier using the same encoder as our model. On SHL the encoder is a reimplementation of FPbiLSTM, whose paper reports 95.1% accuracy and 94.7% F1 (Feature Pyramid biLSTM). On PhysioNet gait it is the CNN-transformer, which reports 87.89% four-stage accuracy (1D-convolutional transformer). The parallel 1D-CNN, reporting 85.3% on UPDRS severity, is a secondary reference (Deep 1D-Convnet). We found no reported result on Lacuna for train-versus-subway or car-versus-bus F1 on SHL, nor for any context-fused model on either dataset, so we will establish these ourselves.

Strongest competitor. The strongest competitor is global fusion: the same encoder with context injected for every class.

  • In the concatenation variant, context is concatenated at the head, following Xue et al..
  • In the TabMixer variant, conditioning is applied throughout the backbone (TabMixer).

This baseline is dangerous because it already uses all the information our model sees.

Ablations that isolate the key ingredients. Two ablations separate routing from the hierarchy itself:

  • Hierarchy without context: the learned tree with backoff, but no context at any node.
  • Ontology versus confusion tree: a hand-designed tree (motorized/non-motorized; PD/control, then stage) in place of the confusion-derived one, testing whether learning the tree matters.

A flat classifier with DeepTag-style top-$k$% abstention to a human is the non-hierarchical abstention baseline.

Fairness and the bar to clear. All models share data splits, the encoder, the context features, and a fixed hyperparameter budget of equal trials per method, and are trained to the same compute. Beating the competitor means the following, all on matched seeds:

  1. Within-pair macro-F1 on confused pairs no worse than global fusion.
  2. Statistically better macro-F1 on non-confused classes.
  3. A dominating hierarchical risk–coverage curve.
  4. A clear advantage when context is randomly dropped for 30–70% of samples, or when labels are cut to 25%.

Why not just…?

Why not just concatenate the context features everywhere? This is the real threat. Satellite availability alone may separate subway from train, and global concatenation will find that. The TFormer ablation suggests extra modalities can hurt unrelated classes. Still, on SHL's large label budget that cost may be negligible, and our matched comparison may show parity. If parity holds even under context dropout and low labels, the honest conclusion is that routing is unnecessary. The remaining contribution would then be backoff decoding and the hierarchical evaluation, and we would report that.

Why not just add a hierarchical or ordinal loss to the motion model, as DeepTag did? The "hierarchy without context" ablation tests exactly this. Hierarchy alone reorganizes errors into the right parent but cannot add information that separates two classes sharing a physical trace. We therefore expect it to raise coverage at the parent level but not leaf accuracy on confused pairs. If it closes the leaf gap too, the context stream is not pulling its weight.

Why not just use gradient-boosted trees on hand-crafted features plus context? Dabiri et al. found that decision trees beat deep models when only 10% of labels were available. Ren et al.'s XGBoost could not separate pedal from electric bikes, because the infrastructure features they shared could not tell those classes apart (Ren et al.). We run XGBoost on hand-crafted features plus context at every label fraction. If it matches our model at low labels, we will recommend it in that regime.

Why not just train a larger signal model? FPbiLSTM is already at 95.1%, and its authors attribute the remaining train–subway errors to identical dynamics. We test this directly by scaling the encoder twofold without context. If the confused-pair F1 does not move, the residual error is physical rather than a capacity limit.

Decisive Checks

Five checks decide whether the claim holds:

  1. Localized gains. Improvement over the signal-only model must concentrate on nodes marked ambiguous. A per-pair breakdown on SHL (train/subway, car/bus, bike/run) and on adjacent gait stages shows whether it does.
  2. Context is actually used. Shuffling context among samples within the same parent node should erase the leaf-level gain. If it does not, the model is exploiting dataset artifacts.
  3. Routing versus fusion operator. Using the same operator globally versus only at ambiguous nodes separates the effect of routing from the effect of the fusion operator.
  4. Robustness. On SHL we cross body positions, training on hand/hip and testing on bag/torso, and we drop context at test time. This checks whether the confusion-derived tree and routing survive shift.
  5. Gait arm. We test whether backoff to "PD, stage ≥2.5" keeps the intermediate stages that clinical studies drop, at an error rate a clinician would accept. This is reported as risk–coverage rather than a single accuracy figure.