TL;DR
This paper introduces Latent HyperNet (LHN), a novel method that enhances Convolutional Neural Networks (CNNs) for human activity recognition (HAR) using wearable sensors. LHN extracts features from multiple layers of a CNN, projects them into a lower-dimensional space, and concatenates them for improved classification, achieving state-of-the-art results without increasing computational cost.
What Problem Does This Paper Address?
Human Activity Recognition (HAR) using wearable sensors is crucial for applications ranging from healthcare to smart environments. While Convolutional Neural Networks (CNNs) have shown great promise in automating feature extraction and classification for HAR, their performance is often limited by the large number of parameters and the scarcity of training data. Traditional CNN approaches primarily rely on features from the final layers, potentially missing valuable information present in earlier layers. Extracting and combining features from all layers directly can lead to prohibitively high dimensionality and computational costs, especially for resource-constrained wearable devices. This paper addresses the challenge of effectively leveraging the rich, multi-scale information from all layers of a CNN for HAR without incurring significant computational overhead.
Methods
The core innovation of this paper is the Latent HyperNet (LHN) approach. LHN works by:
- Iterative Feature Extraction: After each max-pooling layer in a pre-trained CNN, the extracted feature maps are considered.
- Dimensionality Reduction: For each set of feature maps, Partial Least Squares (PLS) is employed to project them onto a lower-dimensional "latent" space. PLS is chosen for its robustness to unbalanced and multi-class problems. This step is crucial for managing the high dimensionality of early layer features.
- Concatenation: The resulting latent features from all considered layers are concatenated.
- Classification: This concatenated feature vector is then fed into a classifier (in this work, a fully connected layer followed by a SoftMax).
A key aspect of LHN is that it does not modify the original CNN architecture or its learned weights. This makes it highly adaptable and easy to integrate with existing CNN models. The authors also propose three novel CNN architectures for HAR, varying in depth and filter configurations, to further evaluate LHN's effectiveness.
Datasets
The authors evaluated their LHN approach on five publicly available HAR datasets:
- USCHAD [19]
- WISDM [20]
- MHEALTH [21]
- UTD-MHAD1 [22]
- UTD-MHAD2 [22]
These datasets vary in sampling rate, number of sensors, and number of activities, allowing for a comprehensive evaluation of LHN's robustness.
Key Insights and Ideas
- Early Layers Hold Valuable Information: The paper strongly supports the hypothesis that features from earlier layers of a CNN, often considered "low-level" or "hyper" features, contain discriminative information that complements the "refined" features from deeper layers.
- Dimensionality Reduction is Key: Directly concatenating features from all layers leads to an unmanageable high-dimensional space. PLS effectively reduces this dimensionality, making the combined features usable without overwhelming the classifier or increasing computational complexity.
- LHN is Architecture-Agnostic: The method's ability to work with pre-trained CNNs without modification is a significant advantage, allowing for easy integration into existing HAR pipelines.
- Multi-Scale Feature Fusion: LHN effectively fuses information from different scales (shallow layers capturing fine details, deep layers capturing abstract patterns) to create a richer representation for activity recognition.
Key Results
- Significant Accuracy Improvements: LHN consistently improved the activity recognition rates of the original CNNs. On average, LHN improved the performance of the proposed ConvNets by 5.1 percentage points compared to state-of-the-art methods. Specific improvements ranged up to 16.70 percentage points on certain datasets.
- State-of-the-Art Performance: The LHN approach, when applied to their proposed CNN architectures and even to an existing state-of-the-art CNN, outperformed existing methods on most datasets. For example, on USCHAD, LHN achieved 83.8% accuracy, and on WISDM, 88.0%.
- No Compromised Computational Cost: Crucially, the paper demonstrates that LHN does not significantly increase the prediction time compared to the original CNN. Statistical tests confirmed that the prediction times are statistically equivalent.
- Importance of Dimensionality Reduction: Removing the dimensionality reduction step (PLS) resulted in a drastic drop in recognition rate (around 30 p.p. on average), highlighting the necessity of this step.
Limitations and Future Work
- Parameter 'c' Selection: While the authors found that
c=19(number of components for PLS) worked best on USCHAD and used it for other datasets, the optimalcmight vary. They suggest that other dimensionality reduction methods like Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA) could also be explored. - Adaptability to Other Domains: The authors express interest in applying LHN to other CNN-based applications, such as image classification, to explore its broader utility.
- Exploring Other Dimensionality Reduction Techniques: Future work will investigate the use of PCA and LDA for building LHNs.
Summary updated