Bridge Inpainting: Teaching Activity Localizers Where Activities End, From Unlabeled Wearable Streams

Wearable motion sensors record continuous streams of acceleration and rotation that people cannot read. To label them, someone has to film participants and line the video up with the sensor clock. Researchers can afford to label a handful of clean examples of "walking" or "sitting," but they almost never label the moments in between: the sit-to-stand, the slowing of a walk, the idle "NULL" stretches. Those in-between moments are where recognizers fail, because the signal changes gradually there and the class boundary is genuinely blurry. We propose to fill exactly those moments with synthetic data. A diffusion model (a generator that learns to turn random noise into realistic samples by removing noise step by step) is trained on large amounts of unlabeled raw sensor streams to fill in a missing middle given what comes before and after. We then give it one labeled snippet of activity A and one of activity B and ask it to generate the bridge between them. The result is a fully labeled continuous timeline with a known boundary, built from a few labeled snippets and no labeled transitions.

The two-sentence version: transitions are abundant in unlabeled recordings but almost never labeled, so we train a generator to learn how real motion moves between states and use it to stitch labeled snippets into realistic, boundary-annotated timelines. These timelines then train a temporal localization model, one that predicts start time, end time and class directly, in a regime where such models normally cannot be trained.

Core Claim

Our claim is that, when labels are scarce, the main obstacle to localizing activities in continuous wearable data is the lack of labeled boundaries, not the lack of labeled interiors. We further claim this obstacle can be removed with a generator that never sees a label. Temporal action localization models such as ActionFormer, TemporalMaxer and TriDet, applied to inertial data in Temporal Action Localization for Inertial-based Human Activity Recognition, remove sliding-window jitter and reported a 26% F1 improvement on SBHAR. That paper motivates our setting. It also constrains it, because those models need fully annotated timelines with exact start and end timestamps.

Generative sensor augmentation does not help with this today. DiffAR builds a step-adaptive conditional diffusion model for WiFi channel-state data and uses it for forecasting, imputation and synthesis. It gains 0.75–3.9% accuracy over unaugmented baselines, but it generates fixed windows, works with full supervision, and ignores transitions. Video2IMU synthesizes inertial signals from video, but it generates whole activities and depends on visually overt motion. The 3D lesion-synthesis GANs, 3D Multi-Conditional GAN-based CT Image Augmentation and CT-Realistic Lung Nodule Simulation, are the closest conceptual precedent. They paint low-contrast lesion boundaries onto healthy scans, but they train under full supervision on volumes.

What is new here is the conditioning. We generate the interface between two labeled endpoints. We learn how to do this only from unlabeled continuous data. Every synthetic sample therefore carries its boundary location by construction.

Research Program

The generator is a one-dimensional diffusion model for imputation, trained on raw tri-axial accelerometer and gyroscope streams from the training subjects' unlabeled recordings. During training we mask a random contiguous gap of variable length and condition on the unmasked left and right context. We reuse DiffAR's idea of feeding coarse context features to early denoising steps and fine high-frequency features to late steps.

We deliberately keep the low-frequency gravity component. Bio-PM high-pass filters at 0.5 Hz to isolate voluntary submovements, and the file notes that this destroys the orientation cues that separate sitting from lying. Postural transitions are largely rotations of the gravity vector, so a bridge generator without gravity would be unable to represent them.

Augmentation proceeds in three steps. First, we sample two labeled snippets of different classes, preferring snippets from the same subject and device. Second, we sample a gap length. Gap lengths come from the widths of low-curvature zones that RECURVE detects on the unlabeled data. RECURVE shows that latent trajectories straighten across boundaries and turn sharply within segments, so its low-curvature zones give an unsupervised estimate of how long real transitions last. Third, we inpaint the gap.

Chaining many such bridges yields long synthetic timelines. We label them as A up to the gap start, B after the gap end, and the gap itself as an ambiguous "A→B transition" region. During that region the localizer's boundary regression receives a soft target spread across the gap instead of a single timestamp. The localizer is the same single-stage architecture used in the temporal-localization inertial paper. It is trained on real labeled snippets plus synthetic timelines. The only thing that changes across conditions is how the bridges are made.

The analysis side asks two questions. Are the synthetic bridges physically plausible? We check that gravity magnitude is preserved and compare their duration and spectral statistics against HAPT's labeled real transitions, which are held out from all training. Do the gains concentrate near boundaries, as the claim predicts, or do they appear uniformly, which would suggest generic regularization?

Baselines

The naive reference point is a sliding-window classifier trained only on the labeled snippets, with median smoothing over predictions. It shows what the label budget buys without any boundary modeling.

The strongest existing label-efficient window method is a few-shot adapter on a time-series foundation model in the style of DeepSenseMoE. DeepSenseMoE adds multi-scale convolutional experts with under 1% extra parameters to MOMENT and reports 1-shot accuracies of 75.1% on HHAR, 83.1% on MotionSense and 74.4% on PAMAP2. Those are window-classification numbers on datasets we do not use for localization, so they are not directly comparable. We will rerun the method on our localization splits, with smoothing, from the same labeled snippets.

The strongest localization competitor is the temporal localization model itself trained only on what is labeled. In the low-label setting that means real snippets arranged as isolated segments, with NULL padding between them. For context, the full-label upper bound is the 26% F1 improvement on SBHAR reported in Temporal Action Localization for Inertial-based Human Activity Recognition. No absolute F1 for SBHAR or the other five datasets is found on Lacuna, so we will rerun the full-label model ourselves as the ceiling.

The ablations that isolate the key ingredient are alternative bridge constructions fed into the identical localizer:

  • Hard cut: A is concatenated directly to B.
  • Crossfade: a linear blend over the same sampled gap length.
  • Retrieval: the unlabeled stretch, from the same subject, whose endpoints best match A's end and B's start, pasted in.
  • Gravity-stripped diffusion: our generator trained on high-pass-filtered data.

Fairness rules apply to every condition. All conditions use the same labeled snippets, the same unlabeled pool, the same number of synthetic timelines, the same localizer architecture and training schedule, and the same hyperparameter search budget. We report generator training cost separately. Our method beats a baseline only if its mean over five label draws times three seeds exceeds the baseline by more than the pooled standard deviation on event-level F1, and on boundary localization error in seconds. The comparison that matters most is against the best of crossfade, retrieval, and pseudo-labeling (described below), not against the naive baseline.

Why not just…?

Why not just crossfade or splice real snippets instead of training a generator? This is the most serious threat. If a linear blend or a hard cut gives the localizer the same boundary supervision, the diffusion model contributes nothing. We have only a physical argument against it. Blending accelerometer values across an orientation change produces an unphysical dip in gravity magnitude. Heuristic transforms in general fail to create realistic intermediate states, the point the lesion-synthesis work makes for images. We do not have direct evidence for inertial bridges. The crossfade and hard-cut ablations settle the question. If crossfade matches diffusion on postural datasets, we will report it as the recommended method and treat the generator as unnecessary.

Why not just retrieve real transitions from the unlabeled stream? Real transitions are already in the unlabeled data, and nearest-endpoint retrieval is cheap and incapable of producing generative artifacts. Its weakness is coverage: rare class pairs and cross-recording pairs may have no good match. The retrieval ablation tests this directly. We additionally break results down by how frequent each class pair is in the unlabeled pool. The generator should win mainly on rare pairs. If it does not win there, retrieval is the better design.

Why not just pseudo-label the unlabeled stream and self-train? One can segment unlabeled recordings with RECURVE, assign each segment to its nearest labeled prototype, and train the localizer on the resulting pseudo-timelines. This uses real data throughout. Its known weakness is that RECURVE produces false boundaries when segments are shorter than its window, which injects label noise exactly at boundaries. We run it as a baseline and also combined with bridge inpainting. If pseudo-labeling alone matches the combination, the generative component is not earning its cost.

Why not just use a stronger window classifier? The temporal-localization inertial paper documents the window-size dilemma and NULL-class jitter. Better window features do not produce boundaries. The DeepSenseMoE-style baseline shows how much of the gap a strong few-shot classifier closes on its own.

Proposed Experiments

The main benchmarks are the six continuous datasets from the temporal-localization inertial paper: SBHAR, Opportunity, Wetlab, WEAR, Hang-Time and RWHAR. All use subject-disjoint splits. We test label budgets of 1, 5 and 20 snippets of 2–5 seconds per class, with all remaining training-subject data unlabeled. Metrics are event-level F1, mean average precision at several temporal-overlap thresholds, and boundary error in seconds. As a secondary metric we report Semantic F1, which gives partial credit to semantically adjacent errors. We use it so that near-miss transition predictions are visible rather than scored as total failures. It is never used for model selection.

The first decisive check is a zero-label transition test on SBHAR/HAPT. That dataset labels six postural transitions, such as sit-to-stand and stand-to-lie. We withhold every transition label and ask whether the localizer, trained only on synthetic A→B bridge regions, recognizes the real transitions above the crossfade and retrieval versions. This is the clearest evidence that the generator learned real transition structure.

The second check stratifies event-level error by distance to the nearest true boundary. Our claim predicts gains concentrated within a few seconds of boundaries and little change in segment interiors.

The third check is the gravity ablation on posture-heavy classes. If keeping gravity matters, the gravity-stripped generator should lose most of its advantage on sit, stand and lie boundaries.

If the first check fails while the second shows only uniform gains, the core claim is wrong. In that case the method is at best a generic augmenter, and we would say so.