Imagine training a neural network to recognize walking using inertial sensor data from ten individuals. The model learns the rhythmic acceleration patterns perfectly. However, when deployed to an eleventh person—who happens to be taller, walks with a faster stride, or wears their smartwatch slightly looser—the model's accuracy plummets.
This is the core challenge of cross-user generalization in wearable sensing. Because human physiology, habits, and environments are highly idiosyncratic, a model trained on a source population encounters a severe domain shift when applied to an unseen target individual. As Yavuz Yarici notes in Subject Invariant Contrastive Learning for Human Activity Recognition, these unique personal signatures often overshadow the universal features of the activity itself, causing models to latch onto subject-specific patterns rather than the underlying physical actions.
To build models that work for everyone, researchers have developed techniques that either strip away individual identity to find invariant features or embrace individual uniqueness through rapid personalization.
Blinding the Model to the User via Domain Adaptation
One way to solve the cross-user problem is to force the neural network to be "blind" to who is performing the action. This is typically achieved through Unsupervised Domain Adaptation (UDA), where the goal is to map sensor data from both source and target users into a shared latent space where user-specific signatures are erased, but activity-specific features remain distinct.
In SALIENCE: An Unsupervised User Adaptation Model for Multiple Wearable Sensors Based Human Activity Recognition, Ling Chen et al. demonstrate that this alignment must happen at multiple granularities. Because different body parts move differently, the authors combine sensor-level "local" alignment with "global" population alignment. This prevents the model from confusing variations in sensor placement with actual variations in movement.
[Source User Data] ---\
--> [Shared Latent Space] --> [Activity Classifier]
[Target User Data] ---/ (User Signatures Erased)
However, standard domain adaptation treats all source users equally, which can be counterproductive if some source users move completely differently than the target user. To address this, the authors introduced a sample-weighting mechanism in SWL-Adapt: An Unsupervised Domain Adaptation Model with Sample Weight Learning for Cross-User Wearable Human Activity Recognition. By using meta-learning to dynamically figure out which source data points are most structurally similar to the target user's distribution, the model prioritizes alignment with the most relevant "kinship" patterns in the training pool.
Furthermore, distribution shifts are often class-dependent; an expert and a novice might look identical while sitting, but vastly different while running. To account for this, Indrajeet Ghosh et al. developed the $\mu$DAR framework, which utilizes class-wise conditional alignment and temporal self-ensembling to ensure that the adaptation process respects the boundaries between different activities, bridging the gap between varying skill levels and physical capabilities.
Embracing Uniqueness via Few-Shot Personalization
Instead of trying to ignore individual differences, an alternative philosophy is to embrace them by quickly customizing a general model to a specific user using a tiny amount of personal data.
The intuition behind this approach lies in how deep neural networks learn representations. The early layers of a Convolutional Neural Network (CNN) typically capture low-level, universal physical properties (like the fundamental frequencies of human motion), while the deeper layers capture high-level semantic combinations which tend to get entangled with individual style.
In Personalized Human Activity Recognition Using Convolutional Neural Networks, Seyed Ali Rokni et al. exploit this architecture through a framework called Transfer Convolutional (TrC). By freezing the early feature-extraction layers of a pre-trained model and fine-tuning only the final classification layer with as few as three labeled samples per activity from the new user, they achieve massive performance boosts over generalized models.
[Raw Sensor Input]
│
▼
┌──────────────────────────────┐
│ Freezed Feature Extractors │ <-- Captures universal physics/frequencies
└──────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Fine-Tuned Classifier Layer │ <-- Personalized to individual style (Few-Shot)
└──────────────────────────────┘
To minimize the burden of collecting these calibration samples, Tom Diethe et al. proposed a hierarchical Bayesian approach. By establishing "community priors" derived from the population, the model can make highly accurate baseline predictions for a new user before seeing a single label. It then uses active learning to selectively ask the user to label only the most ambiguous movements, rapidly personalizing the system with minimal human intervention.
Decentralized Personalization via Federated Meta-Learning
While personalization is highly effective, wearable sensor data is deeply intimate, raising significant privacy concerns if raw data is uploaded to a central server for model tuning. This constraint has driven the field toward decentralized, privacy-preserving adaptation frameworks.
The foundational architecture for this is explored by Yiqiang Chen et al. in FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare. Instead of aggregating data, FedHealth aggregates model updates across a network of users to build a robust global model. This global model is then downloaded to individual edge devices, where it undergoes local transfer learning to adapt to the specific user's gait, speed, and physical constraints without their raw data ever leaving the device.
Central Server: [Aggregate Global Model]
▲ │
Model Updates │ ▼ Global Weights
│ │
Edge Devices: [User A Device] [User B Device]
│ │
▼ ▼
(Local Tuning) (Local Tuning)
To optimize this pipeline, Chenglin Li et al. introduced Meta-HAR: Federated Representation Learning for Human Activity Recognition. Rather than training the global model to perform well on average, Meta-HAR uses federated meta-learning to train the global model specifically to be easy to adapt.
By framing the learning process as a multi-task meta-problem, the global model learns a highly flexible signal embedding space. When a new user adopts the device, the local model requires only a fraction of the gradient steps to adjust to their unique biomechanical signature, successfully resolving the tension between universal generalization and individual personalization.