Cross-Sensor Scale Selection for Subject-Robust Activity Recognition

People do the same activity at different speeds. One person's stride lasts 0.9 seconds and another's lasts 1.3. A convolutional network that reads a wrist accelerometer through a fixed set of temporal window sizes therefore sees "walking" at a different scale for every user. This is a plausible reason deep activity recognizers lose ground to hand-crafted features when tested on people they have never seen. Hand-crafted pipelines quietly normalize for tempo: features such as dominant frequency and spectral entropy describe the rhythm of the signal, not its absolute timing. We propose to recover that invariance without hand engineering. A bank of dilated convolutions (filters whose taps are spaced apart, so one small kernel covers a long stretch of time) reads each sensor at many time scales in parallel. A second sensor on a different body part then decides, moment by moment, which of those scales to listen to. In a typical case the ankle IMU, which measures cadence cleanly, tells the wrist and chest streams which time scale currently corresponds to "one step."

The cross-modal attention here runs over scales rather than over time steps. That changes two things. Its cost grows linearly with sequence length instead of quadratically. It also gives the fusion a job that is physically meaningful and checkable: the selected scale should track the subject's measured cadence. The test bed is strict leave-one-subject-out evaluation on PAMAP2 and OPPORTUNITY, which is where the case against deep features is strongest. The decisive comparison is against the simplest hand-engineered fix, which is to estimate cadence and resample the window.

Core Claim

Much of the subject-to-subject gap in wearable activity recognition comes from differences in tempo. A network can absorb this gap with a dilated multi-scale filter bank combined with a gate that picks the scale using a different sensor. Under leave-one-subject-out testing, this combination should do three things. It should match or beat a Random Forest on hand-crafted features. It should beat the same backbone fused by concatenation or by time-axis cross-attention. It should degrade much less when test subjects are artificially sped up or slowed down. The cross-sensor part is a falsifiable claim, not decoration. If the scale chosen by a sensor's own features works as well, the contribution shrinks to "attention over scales," and we will report it as such.

Relation to Prior Work

Three papers motivate the problem. Rehman et al. showed on PAMAP2 that a Random Forest falls from 97% under random k-fold to 89% under leave-one-subject-out. They also showed that hand-crafted features beat raw input by about 30% for that same Random Forest. That second comparison is between feature sets for a shallow classifier, not between hand features and deep networks. We cite it as evidence that hand features carry invariances raw windows lack, not as evidence that deep models lose by 30%. Kalabakov et al. found DeepConvLSTM only roughly matches a Random Forest on about 2,400 hand-crafted features, and that transfer works best when just the first convolutional layer is frozen. Both papers say the gap sits in low-level temporal filtering. Speed variation is named as a failure of fixed windows and single-scale kernels by Rahmattalabi et al. and Li et al..

The closest system is CMD-HAR (Liu et al.). It groups sensor channels and uses self- and cross-attention to split features into spatial, temporal and shared parts. It adds gradient balancing so the accelerometer does not dominate training, and reports 86.23% accuracy and 0.91 G-mean on OPPORTUNITY. CMD-HAR attends over time and channels at a single convolutional scale. We attend over scales and do not use a disentanglement loss.

THAT (Li et al.) combines parallel kernels of different sizes with self-attention for WiFi sensing, without cross-sensor fusion. We keep THAT's multi-scale idea but make scale selection depend on another modality. MMTSA (Wang et al.) fuses IMU and video through image conversion. We stay in 1D and avoid video.

Several papers supply the method. PSConv (Li et al.) and CFPNet (Lou and Loew) show that dilation rates can be spread across channel groups at little or no parameter cost while avoiding gridding, meaning holes left by sparse sampling. We port this to 1D and keep a dense dilation-1 branch. DCLS (Khalfaoui-Hassani et al.) learns non-integer tap positions by interpolation. SiGMA (Lee et al.) argues that integer scale steps leave an irreducible "quantization gap." Together they justify adding fractional dilations to the bank, so the scale grid is fine enough to follow continuous tempo changes. The gate comes from gated cross-attention (Kim and Shin). That work showed ungated cross-attention can overwrite a stream's own features, and that mixing back the original representation prevents this.

Research Program

Each sensor stream $m$ (for PAMAP2: wrist, chest and ankle IMUs, plus heart rate) passes through a shared stem. It then enters a bank of $S$ depthwise-separable 1D convolutions with dilations spaced roughly geometrically from 1 to about 16. Fractional values are realized by linear interpolation of taps. Each branch produces features $h^{m}_{s}(t)$ at scale $s$ and time $t$.

For a target stream $m$ and a guide stream $n$, the guide emits a query $q^{n}(t)$. Each target scale emits a key $k^{m}_{s}(t)$. The fused feature is

$$z^{m}(t) = g^{m}(t)\sum_{s=1}^{S}\alpha^{m}_{s}(t)\,h^{m}_{s}(t) + \big(1-g^{m}(t)\big)\,h^{m}_{1}(t), \qquad \alpha^{m}_{s}(t)=\mathrm{softmax}_{s}\!\left(\frac{q^{n}(t)^{\top}k^{m}_{s}(t)}{\sqrt{d}}\right).$$

Here $\alpha^{m}_{s}(t)$ is the attention weight on scale $s$. $d$ is the key dimension. $g^{m}(t)\in[0,1]$ is a learned gate that preserves the finest-scale feature $h^{m}_{1}$, following Kim and Shin. The softmax runs over $S$ scales per time step, so the cost is $O(TS)$ rather than the $O(T^2)$ of time-axis cross-attention. Which sensor guides which is learned with a small learned weighting over guides, so we do not hard-code "ankle guides wrist." The fused streams are pooled and classified by a light temporal head. The overall model is kept at or below CMD-HAR-scale latency on embedded hardware.

The analysis program asks whether the learned scale is physically meaningful. Per subject and window, we compute the attention-weighted dilation $\bar d(t)=\sum_s \alpha_s(t)\,d_s$. We correlate it with the cadence estimated independently from the ankle signal's dominant frequency. A strong negative correlation (faster cadence, smaller selected dilation) would support the tempo-normalization account. A flat relationship would mean the gains, if any, come from something else.

Baselines

The naive reference is DeepConvLSTM (Kalabakov et al.) with all channels stacked at the input. A second naive reference is the unification-layer CNN of Yang et al., which concatenates per-sensor feature maps. These show what plain fusion at a single scale achieves. Any proposal that cannot beat them clearly under leave-one-subject-out testing has nothing to say.

The strongest classical competitor is a Random Forest on time- and frequency-domain hand-crafted features. We follow the 50-feature selection of Rehman et al., who report 89% leave-one-subject-out accuracy on PAMAP2. It is the competitor that directly embodies the invariance we claim to learn. Beating it means exceeding its accuracy and macro-F1 on identical windows and identical held-out subjects, averaged over all folds. A tie at 89% would be a meaningful result in its own right.

The strongest deep competitor is CMD-HAR (Liu et al.), which reports 86.23% accuracy and 0.91 G-mean on OPPORTUNITY. It already handles sensor dominance with cross-attention and gradient balancing, so it is the right test of whether attending over scales adds anything beyond attending over time and channels. Its published number may not use leave-one-subject-out splits. We therefore reimplement it, and run it on our splits with the same window length, the same input channels, an equal hyperparameter search budget of 30 trials per method on inner-fold validation subjects, and matched training epochs. We report the published figure only as a sanity check.

We also run THAT's multi-scale-kernel transformer from windofshadow/THAT, adapted to IMU input. It represents multi-scale filtering paired with self-attention only. For OPPORTUNITY, the b-LSTM of Hammerla et al. is a further reference at a reported 0.745 mean F1, again under its own protocol.

Three ablations isolate the key ingredient. The first keeps the identical filter bank but replaces scale attention with concatenation of all scales. The second keeps scale attention but takes the query from the target stream itself instead of from another sensor; this is the critical one. The third uses an integer-only dilation bank, which tests whether fractional scales matter.

Why not just…?

Why not just use a Random Forest on hand-crafted features? Under leave-one-subject-out testing it is a real threat. The 89% PAMAP2 figure of Rehman et al. may be hard to beat on a nine-subject dataset. Its weaknesses lie elsewhere: features must be re-engineered when sensor placement changes, and on transportation-mode data Bhattacharya et al. found hand features reached 67.9% F1 against 79.9% for learned sparse codes. We will report honestly if the forest wins on PAMAP2. The tempo-shift test below is where we expect the clearest separation, because fixed frequency features shift when a subject is sped up.

Why not just estimate cadence and resample each window to a canonical tempo? This is the most dangerous alternative. It is one line of signal processing and directly implements our hypothesis. We include it as a baseline: dominant-frequency resampling feeding the same backbone. It may well match our model on gait-dominated PAMAP2 classes. Our expected advantages are in non-periodic OPPORTUNITY gestures, where no cadence exists to estimate, and in windows mixing two activities. If resampling matches us on both datasets, the right conclusion is that tempo normalization is the active ingredient and the learned version is unnecessary.

Why not just widen kernels or apply time-warp augmentation? Larger or parallel standard kernels, as in Li et al., give multi-scale coverage at higher parameter cost. Augmentation teaches tolerance to speed changes rather than adapting to them. Both are cheap, so we run a parameter-matched multi-kernel backbone and a time-warp-augmented DeepConvLSTM, each with an augmentation range equal to our tempo-shift test range. If augmentation closes the gap, the scale-selection mechanism has not earned its place.

Proposed Experiments

All experiments use strict leave-one-subject-out splits. That means nine folds on PAMAP2 with heart rate as an optional guide stream, and four on OPPORTUNITY body-worn IMUs. A secondary check uses USC-HAD, following CMD-HAR's dataset list. Metrics are accuracy, macro-F1 and G-mean across folds, with per-fold paired comparisons and a sign test across subjects, plus parameter count and latency on a Raspberry-class CPU.

The first decisive check is the tempo-shift curve. Held-out subjects are resampled by factors from 0.75 to 1.33. We plot macro-F1 against the factor for every method. The proposal predicts a flatter curve than every fixed-scale baseline and the Random Forest. It should be at least as flat as time-warp augmentation, despite never seeing warped training data.

The second decisive check is the cross-sensor versus self-query ablation, reported separately for periodic and non-periodic classes. The third is the correlation between the selected dilation and independently measured cadence across held-out subjects.

Three outcomes would falsify the idea. Cadence resampling matching our model on both datasets would. So would self-query matching cross-sensor query. So would the absence of any cadence correlation. Any one of these would redirect the work. If all three come out as predicted, we would have a learned replacement for the tempo normalization that hand-crafted features provide implicitly, with a direct check that it is doing what it claims.