A Framework of Combining Short-Term Spatial/Frequency Feature Extraction and Long-Term IndRNN for Activity Recognition

Beidi Zhao, Shuai Li, Yanbo Gao, Chuankun Li, Wanqing Li

Abstract

Smartphone-based human activity recognition (HAR) faces significant challenges due to high sensor sampling rates, large intra-class distances (varying phone placements), and small inter-class distances (similar transportation modes). We propose a framework that combines short-term spatial and frequency feature extraction with a long-term Independently Recurrent Neural Network (IndRNN). By extracting periodic characteristics in both spatial and frequency domains and leveraging IndRNN's ability to capture long-term patterns, our method achieves 80.72% accuracy on the Sussex-Huawei Locomotion (SHL) dataset, outperforming existing single-model methods.

Introduction

The ubiquity of smartphones makes them ideal for tracking daily activities and transportation modes. However, building robust HAR models must overcome three primary obstacles:

  1. Long-range temporal dependencies: High sampling rates (e.g., 100Hz) create long sequences where standard RNNs or CNNs often lose context.
  2. Large intra-class variance: Sensor signals for the same activity (e.g., walking) vary significantly depending on whether the phone is in a pocket, bag, or hand.
  3. Small inter-class variance: Activities like riding a train versus a subway produce nearly identical sensor profiles, making them difficult to distinguish.

Proposed Methodology

The proposed framework addresses these issues through a multi-stage pipeline designed to handle long sequences and generalize across different usage scenarios.

The overall framework of the proposed method, illustrating the flow from raw sensor data to final predictions.

Preprocessing and Feature Extraction

Raw sensor data is first "derotated" using quaternion orientation data to align accelerometer and magnetometer readings with a global North-East-Down coordinate system. This ensures consistency regardless of the phone's physical orientation relative to the user.

To manage the high-frequency data, the system segments 5-second windows into 21 overlapping 1-second sub-windows. For each sub-window, the model extracts:

  • Spatial Features: Statistical measures like mean and variance.
  • Frequency Features: FFT amplitude spectrums to capture rhythmic movement patterns.
  • Temporal Differences: Augmenting features with changes over time to capture motion dynamics.

Long-Term Modeling with Dense IndRNN

The sequence of extracted features is processed by a deep, dense Independently Recurrent Neural Network (IndRNN). Unlike traditional LSTMs, IndRNN neurons are independent within a layer, which mitigates gradient vanishing/exploding issues and allows for much deeper architectures. These layers are organized into "Dense blocks" to maximize feature reuse across temporal scales.

Location-Aware Transfer Learning

To handle unknown phone placements and user-specific movement styles, the framework employs a two-step post-processing strategy. First, it classifies the phone's location into broad groups (e.g., Hand/Bag vs. Hips/Torso). Second, it uses fused transfer learning to fine-tune the model on a small subset of target-user data, significantly improving generalization.

Experimental Results

The framework was evaluated on the Sussex-Huawei Locomotion (SHL) dataset, which includes 8 activity classes across 4 body locations.

FFT amplitude spectrums for different activities, demonstrating how frequency-domain features help distinguish between rhythmic movements.

The Dense IndRNN framework achieved an 80.72% F1-score, surpassing single-model baselines like XGBoost (77.9%) and CNN+LSTM hybrids (52.8%). Ablation studies confirmed that the dense architecture improved performance by 2.3% over a standard IndRNN, while feature augmentation added another 2.2%. Transfer learning proved critical for user independence; for instance, accuracy for the "Run" class jumped from 43% to 94% after fine-tuning.

Conclusion

By combining short-term frequency analysis with deep long-term recurrent modeling, this framework effectively bridges the gap between raw sensor data and complex human behaviors. While it excels at distinguishing physical locomotion, future work will focus on better differentiating motor-powered transport modes (e.g., bus vs. train) to further close the performance gap with heavy ensemble methods while maintaining a single core architecture.

Summary updated

Abstract

Smartphone sensors based human activity recognition is attracting increasing interests nowadays with the popularization of smartphones. With the high sampling rates of smartphone sensors, it is a highly long-range temporal recognition problem, especially with the large intra-class distances such as the smartphones carried at different locations such as in the bag or on the body, and the small inter-class distances such as taking train or subway. To address this problem, we propose a new framework of combining short-term spatial/frequency feature extraction and a long-term Independently Recurrent Neural Network (IndRNN) for activity recognition. Considering the periodic characteristics of the sensor data, short-term temporal features are first extracted in the spatial and frequency domains. Then the IndRNN, which is able to capture long-term patterns, is used to further obtain the long-term features for classification. In view of the large differences when the smartphone is carried at different locations, a group based location recognition is first developed to pinpoint the location of the smartphone. The Sussex-Huawei Locomotion (SHL) dataset from the SHL Challenge is used for evaluation. An earlier version of the proposed method has won the second place award in the SHL Challenge 2020 (the first place if not considering multiple models fusion approach). The proposed method is further improved in this paper and achieves 80.72$\%$ accuracy, better than the existing methods using a single model.
BibTeX
@article{zhao2020framework,
  author = {Beidi Zhao and Shuai Li and Yanbo Gao and Chuankun Li and Wanqing Li},
  title = {{A Framework of Combining Short-Term Spatial/Frequency Feature Extraction and Long-Term IndRNN for Activity Recognition}},
  journal = {Sensors},
  year = {2020},
  doi = {10.3390/s20236984},
}
Figure