Federated Meta-Learning with Attention for Diversity-Aware Human Activity Recognition

Qiang Shen, Haotian Feng, Rui Song, Donglei Song, Hao XuSensors 2023 · DOI

TL;DR

This paper introduces DivAR, a federated meta-learning framework for human activity recognition (HAR) that tackles the challenges of data privacy and individual diversity. By clustering users based on their characteristics and employing attention mechanisms within a federated learning setup, DivAR achieves robust and personalized activity recognition, even for new users.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) using smartphone sensors holds immense potential for applications like smart homes and health monitoring. However, two major hurdles impede its widespread adoption:

  1. Data Privacy: Deep learning models for HAR require vast amounts of sensor data, which are sensitive and cannot be easily shared with a central server due to privacy concerns.
  2. Individual Diversity: People exhibit diverse behavioral patterns and lifestyles, leading to significant variations in sensor data even for the same activity. This "heterogeneity" makes it difficult for a single model to perform well across all users, especially for new users ("cold-start" problem).

Existing solutions often struggle to balance privacy with the need for personalization and struggle to account for the nuanced differences between individuals.

Methods

DivAR proposes a novel approach combining Federated Learning and Meta-Learning with an Attention Mechanism to address these challenges:

  • Federated Learning: This allows models to be trained locally on user devices without sharing raw data. Only model updates are sent to a central server, preserving privacy.
  • Meta-Learning (specifically, a Reptile-like approach): This enables the model to "learn to learn." A central meta-model learns general feature representations that can be quickly adapted to new tasks or users.
  • Individual Clustering: Before meta-learning, individuals are grouped into clusters based on their social factors (e.g., personality, mental state) and behavioral patterns. This allows for more targeted learning within each group.
  • Attention Mechanism: Within each decentralized client model, a CNN-based attention module is employed. This module learns to selectively focus on the most relevant sensor features for a specific cluster, effectively extracting individual-specific features from the globally learned representations.

The architecture involves a central server that maintains a shared embedding network and decentralized client models that learn cluster-specific features using the attention mechanism.

Datasets

The authors created two new datasets for their experiments:

  • Dataset 1 & Dataset 2: These datasets consist of sensor readings from naturally used smartphones, annotated with activity information. They were collected from multiple individuals in real-life environments. Crucially, these datasets also include social characteristics (personality, mental health, behavior patterns) collected via questionnaires, which are used for individual clustering. These datasets are described as "multi-individual heterogeneous."

Key Insights and Ideas

  • Clustering is Key: Grouping individuals based on their characteristics significantly improves the model's ability to handle heterogeneity and generalize across users. The paper shows that DivAR-cluster (with clustering) consistently outperforms DivAR-individual (without clustering).
  • Attention for Personalization: The attention mechanism is vital for extracting individual-specific features from shared representations, allowing the model to adapt to diverse behavioral patterns.
  • Beyond Basic Features: Incorporating social and behavioral factors for clustering is more effective than relying solely on basic demographic information or even just behavioral type.
  • Federated Meta-Learning Synergy: The combination of federated learning for privacy and meta-learning for rapid adaptation is a powerful paradigm for personalized HAR in real-world, distributed settings.

Key Results

  • Superior Performance: DivAR consistently outperformed baseline methods (Random Forest, DeepSense, AttenSense, Meta-HAR) in terms of accuracy and macro F1-score across three datasets (two custom, one mixed).
  • Significant Gains: DivAR-cluster achieved performance gains of up to 5.50% in accuracy and 9.61% in macro F1-score compared to its individual-focused counterpart.
  • Effective Cold-Start Adaptation: DivAR demonstrated strong performance in adapting to new users, outperforming Meta-HAR by effectively capturing feature-level discrepancies.
  • Computational Efficiency: DivAR generally requires fewer training epochs to achieve its best performance compared to other models.

Limitations and Future Work

  • Computational Resources: The paper acknowledges that deploying deep learning models on edge devices like smartphones requires significant computational resources, and models may need compression for smaller devices.
  • Unsupervised Clustering: The authors suggest that more unsupervised methods for clustering users could be explored to reduce reliance on collecting explicit user information.
  • Future Work: The authors plan to explore additional social factors for clustering and conduct correlation analysis to enhance model explainability.

Summary updated

Abstract

The ubiquity of smartphones equipped with multiple sensors has provided the possibility of automatically recognizing of human activity, which can benefit intelligent applications such as smart homes, health monitoring, and aging care. However, there are two major barriers to deploying an activity recognition model in real-world scenarios. Firstly, deep learning models for activity recognition use a large amount of sensor data, which are privacy-sensitive and hence cannot be shared or uploaded to a centralized server. Secondly, divergence in the distribution of sensory data exists among multiple individuals due to their diverse behavioral patterns and lifestyles, which contributes to difficulty in recognizing activity for large-scale users or 'cold-starts' for new users. To address these problems, we propose DivAR, a diversity-aware activity recognition framework based on a federated Meta-Learning architecture, which can extract general sensory features shared among individuals by a centralized embedding network and individual-specific features by attention module in each decentralized network. Specifically, we first classify individuals into multiple clusters according to their behavioral patterns and social factors. We then apply meta-learning in the architecture of federated learning, where a centralized meta-model learns common feature representations that can be transferred across all clusters of individuals, and multiple decentralized cluster-specific models are utilized to learn cluster-specific features. For each cluster-specific model, a CNN-based attention module learns cluster-specific features from the global model. In this way, by training with sensory data locally, privacy-sensitive information existing in sensory data can be preserved. To evaluate the model, we conduct two data collection experiments by collecting sensor readings from naturally used smartphones annotated with activity information in the real-life environment and constructing two multi-individual heterogeneous datasets. In addition, social characteristics including personality, mental health state, and behavior patterns are surveyed using questionnaires. Finally, extensive empirical results demonstrate that the proposed diversity-aware activity recognition model has a relatively better generalization ability and achieves competitive performance on multi-individual activity recognition tasks.
BibTeX
@inproceedings{shen2023federated,
  author = {Qiang Shen and Haotian Feng and Rui Song and Donglei Song and Hao Xu},
  title = {{Federated Meta-Learning with Attention for Diversity-Aware Human Activity Recognition}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23031083},
}
Figure