Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition

Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, Shonali KrishnaswamyIJCAI 2015

Introduction

Human Activity Recognition (HAR) is a fundamental component of ubiquitous computing, enabling applications in healthcare, fitness, and smart environments. Traditionally, HAR systems have relied on hand-crafted features—such as statistical measures or frequency-domain transforms—extracted from sensor data and classified using shallow models like Support Vector Machines (SVMs). However, these heuristic-based features often fail to capture the complex, hierarchical nature of human movement and require significant domain expertise. This paper proposes a deep Convolutional Neural Network (CNN) architecture designed to automatically learn discriminative features from multichannel time-series data, addressing the limitations of manual feature engineering.

Methodology

The proposed architecture adapts the principles of 2D CNNs, widely successful in image recognition, to 1D temporal sensor data. The network consists of five primary stages:

  1. Temporal Convolution and Subsampling: Instead of spatial filters, the model employs 1D filters that convolve along the time dimension for each sensor channel. This process captures local temporal patterns characteristic of atomic movements.
  2. Non-linearity and Normalization: Rectified Linear Units (ReLU) are used to introduce non-linearity, while normalization layers stabilize the learning process.
  3. D-sensor Unification Layer: A critical innovation is the "unification layer." Unlike standard CNNs that assume a unified input grid, HAR data comes from multiple independent sensors. This layer parametrically concatenates feature maps from all $D$ sensors into a single representation before the final classification stages.
  4. Fully Connected Layers: These layers integrate the unified features to form high-level abstractions of activities.
  5. Softmax Classification: The final layer outputs a probability distribution across the target activity classes.
The proposed CNN architecture for multichannel time-series data, featuring temporal convolutions and a sensor unification layer.

To account for the temporal continuity of human actions, a post-processing smoothing step (e.g., a low-pass filter or majority voting) is applied to the network's predictions, reducing unnatural "flickering" between activity states.

Experimental Setup

The model was evaluated on two benchmark datasets:

  • Opportunity Activity Recognition: A dataset featuring 72 sensors (body-worn and ambient) tracking 18 classes of daily activities, such as "opening a fridge" or "drinking from a cup."
  • Hand Gesture Dataset: Focused on 12 distinct hand movements using accelerometers and gyroscopes.

The CNN was compared against several baselines, including SVMs, k-Nearest Neighbors (k-NN), and unsupervised Deep Belief Networks (DBNs).

Results and Discussion

The CNN consistently outperformed traditional and unsupervised models across both datasets. On the Opportunity dataset, the CNN achieved a significant improvement in accuracy and F-measure compared to state-of-the-art baselines.

Performance comparison (AF: Average F-measure, NF: Normalized F-measure, AC: Accuracy) on the Opportunity dataset across three subjects.

Key findings include:

  • Supervised Feature Learning: The supervised nature of the CNN allows it to learn features specifically tuned for classification, outperforming the unsupervised feature extraction of DBNs.
  • Robustness to Noise: The model showed superior performance in distinguishing actual activities from the "Null" class (background noise), although the dominance of the Null class in training data remains a challenge.
  • Computational Efficiency: Inference is highly efficient, with the model capable of processing 56 instances per second on a standard CPU, making it suitable for real-time applications.

Conclusion

This research demonstrates that deep CNNs can effectively automate feature extraction for multichannel time-series data in HAR. By utilizing a novel unification layer and temporal convolutions, the proposed model achieves superior performance over traditional hand-crafted methods. Future work will focus on addressing class imbalance and optimizing hyperparameters for diverse sensor configurations.

Summary updated

BibTeX
@inproceedings{yang2015deep,
  author = {Jianbo Yang and Minh Nhut Nguyen and Phyo Phyo San and Xiaoli Li and Shonali Krishnaswamy},
  title = {{Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure