Latent HyperNet: Exploring the Layers of Convolutional Neural Networks

Artur Jordão, Ricardo Barbosa Kloss, William Robson Schwartz

TL;DR

This paper introduces Latent HyperNet (LHN), a novel method that enhances Convolutional Neural Networks (CNNs) for human activity recognition (HAR) using wearable sensors. LHN extracts features from multiple layers of a CNN, projects them into a lower-dimensional space, and concatenates them for improved classification, achieving state-of-the-art results without increasing computational cost.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) using wearable sensors is crucial for applications ranging from healthcare to smart environments. While Convolutional Neural Networks (CNNs) have shown great promise in automating feature extraction and classification for HAR, their performance is often limited by the large number of parameters and the scarcity of training data. Traditional CNN approaches primarily rely on features from the final layers, potentially missing valuable information present in earlier layers. Extracting and combining features from all layers directly can lead to prohibitively high dimensionality and computational costs, especially for resource-constrained wearable devices. This paper addresses the challenge of effectively leveraging the rich, multi-scale information from all layers of a CNN for HAR without incurring significant computational overhead.

Methods

The core innovation of this paper is the Latent HyperNet (LHN) approach. LHN works by:

  1. Iterative Feature Extraction: After each max-pooling layer in a pre-trained CNN, the extracted feature maps are considered.
  2. Dimensionality Reduction: For each set of feature maps, Partial Least Squares (PLS) is employed to project them onto a lower-dimensional "latent" space. PLS is chosen for its robustness to unbalanced and multi-class problems. This step is crucial for managing the high dimensionality of early layer features.
  3. Concatenation: The resulting latent features from all considered layers are concatenated.
  4. Classification: This concatenated feature vector is then fed into a classifier (in this work, a fully connected layer followed by a SoftMax).

A key aspect of LHN is that it does not modify the original CNN architecture or its learned weights. This makes it highly adaptable and easy to integrate with existing CNN models. The authors also propose three novel CNN architectures for HAR, varying in depth and filter configurations, to further evaluate LHN's effectiveness.

Datasets

The authors evaluated their LHN approach on five publicly available HAR datasets:

  • USCHAD [19]
  • WISDM [20]
  • MHEALTH [21]
  • UTD-MHAD1 [22]
  • UTD-MHAD2 [22]

These datasets vary in sampling rate, number of sensors, and number of activities, allowing for a comprehensive evaluation of LHN's robustness.

Key Insights and Ideas

  • Early Layers Hold Valuable Information: The paper strongly supports the hypothesis that features from earlier layers of a CNN, often considered "low-level" or "hyper" features, contain discriminative information that complements the "refined" features from deeper layers.
  • Dimensionality Reduction is Key: Directly concatenating features from all layers leads to an unmanageable high-dimensional space. PLS effectively reduces this dimensionality, making the combined features usable without overwhelming the classifier or increasing computational complexity.
  • LHN is Architecture-Agnostic: The method's ability to work with pre-trained CNNs without modification is a significant advantage, allowing for easy integration into existing HAR pipelines.
  • Multi-Scale Feature Fusion: LHN effectively fuses information from different scales (shallow layers capturing fine details, deep layers capturing abstract patterns) to create a richer representation for activity recognition.

Key Results

  • Significant Accuracy Improvements: LHN consistently improved the activity recognition rates of the original CNNs. On average, LHN improved the performance of the proposed ConvNets by 5.1 percentage points compared to state-of-the-art methods. Specific improvements ranged up to 16.70 percentage points on certain datasets.
  • State-of-the-Art Performance: The LHN approach, when applied to their proposed CNN architectures and even to an existing state-of-the-art CNN, outperformed existing methods on most datasets. For example, on USCHAD, LHN achieved 83.8% accuracy, and on WISDM, 88.0%.
  • No Compromised Computational Cost: Crucially, the paper demonstrates that LHN does not significantly increase the prediction time compared to the original CNN. Statistical tests confirmed that the prediction times are statistically equivalent.
  • Importance of Dimensionality Reduction: Removing the dimensionality reduction step (PLS) resulted in a drastic drop in recognition rate (around 30 p.p. on average), highlighting the necessity of this step.

Limitations and Future Work

  • Parameter 'c' Selection: While the authors found that c=19 (number of components for PLS) worked best on USCHAD and used it for other datasets, the optimal c might vary. They suggest that other dimensionality reduction methods like Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA) could also be explored.
  • Adaptability to Other Domains: The authors express interest in applying LHN to other CNN-based applications, such as image classification, to explore its broader utility.
  • Exploring Other Dimensionality Reduction Techniques: Future work will investigate the use of PCA and LDA for building LHNs.

Summary updated

Abstract

Since Convolutional Neural Networks (ConvNets) are able to simultaneously learn features and classifiers to discriminate different categories of activities, recent works have employed ConvNets approaches to perform human activity recognition (HAR) based on wearable sensors, allowing the removal of expensive human work and expert knowledge. However, these approaches have their power of discrimination limited mainly by the large number of parameters that compose the network and the reduced number of samples available for training. Inspired by this, we propose an accurate and robust approach, referred to as Latent HyperNet (LHN). The LHN uses feature maps from early layers (hyper) and projects them, individually, onto a low dimensionality (latent) space. Then, these latent features are concatenated and presented to a classifier. To demonstrate the robustness and accuracy of the LHN, we evaluate it using four different network architectures in five publicly available HAR datasets based on wearable sensors, which vary in the sampling rate and number of activities. We experimentally demonstrate that the proposed LHN is able to capture rich information, improving the results regarding the original ConvNets. Furthermore, the method outperforms existing state-of-the-art methods, on average, by 5.1 percentage points.
BibTeX
@inproceedings{jordao2018latent,
  author = {Artur Jordão and Ricardo Barbosa Kloss and William Robson Schwartz},
  title = {{Latent HyperNet: Exploring the Layers of Convolutional Neural Networks}},
  booktitle = {IJCNN},
  year = {2018},
  url = {https://openreview.net/forum?id=0nVPu33si0},
}
Figure