DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition

Zhan Yang, Osolo Ian Raymond, Chengyuan Zhang, Ying Wan, Jun Long

TL;DR

This paper introduces DFTerNet, a novel approach to human activity recognition (HAR) that significantly reduces the computational and memory footprint of deep learning models. By quantizing network weights and activations to 2-bit values and employing a dynamic sensor fusion strategy, DFTerNet achieves near full-precision accuracy with substantial speedups and memory savings, making advanced HAR feasible on resource-constrained devices.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (DCNNs) have shown great promise in Human Activity Recognition (HAR), but their high computational and memory demands make them impractical for deployment on portable devices like smartphones and VR/AR systems. This limitation hinders the development of sophisticated AI-powered applications, such as realistic interactive games, that rely on real-time activity recognition. Furthermore, existing HAR models often treat data from different sensors (e.g., IMUs on various body parts) uniformly, failing to account for the fact that different activities might benefit from different sensor combinations or weighting.

Methods

The core of DFTerNet lies in two key innovations:

  • 2-bit Quantization: The authors propose a quantization scheme that constrains both weights and activations to 2-bit values: {-0.5, 0, 0.5}. This is achieved through an "elastic scale parameter" (ε-Q) that allows for flexible discretization. This quantization significantly reduces memory usage and enables the replacement of costly floating-point operations with more efficient bitwise computations.
  • Dynamic Fusion Strategy: Instead of treating all sensor inputs equally, DFTerNet dynamically adjusts the contribution of different sensor modalities based on the activity being recognized. This is achieved by learning fusion weights that are sampled from a Bernoulli distribution, effectively "dropping out" or down-weighting less relevant sensor signals for specific activities. This approach aims to improve accuracy by focusing on the most informative sensor data.

Datasets

The proposed DFTerNet was evaluated on three publicly available HAR datasets:

  • OPPORTUNITY: A widely used dataset with four subjects performing 17 daily activities, collected using numerous wireless IMUs.
  • PAMAP2: Another comprehensive dataset featuring 12 activities (including household chores and exercises) collected from 9 subjects using IMUs and a heart rate monitor.
  • UniMiB-SHAR: A dataset comprising 17 activity classes (including safety and dangerous activities) collected from 30 healthy subjects using a smartphone's 3D accelerometer.

Key Insights and Ideas

  • Quantization is Key for Portability: The paper strongly reinforces the idea that aggressive quantization (down to 2-bit) is crucial for enabling complex DCNNs on devices with limited computational power and memory.
  • Activity-Specific Sensor Fusion Matters: Treating sensor inputs uniformly is suboptimal. A dynamic fusion strategy that adapts to the specific activity being recognized can significantly improve performance by leveraging the most relevant sensor data.
  • Trade-off Between Accuracy and Efficiency: DFTerNet demonstrates that it's possible to achieve a favorable trade-off, obtaining performance very close to full-precision models while drastically reducing resource requirements.
  • Sparsity as a Regularizer: The visualization of zero values in network layers suggests that quantization can induce sparsity, which may act as a beneficial regularizer, preventing overfitting.

Key Results

  • Accuracy: DFTerNet achieved performance comparable to full-precision models, with improvements of up to ~5% over baseline models on the OPPORTUNITY and PAMAP2 datasets when using the dynamic fusion strategy.
  • Speedup: The proposed method achieves approximately 9x acceleration on CPUs.
  • Memory Saving: DFTerNet requires approximately 11x less memory compared to its full-precision counterpart.
  • Efficiency: The use of bitwise operations and Hamming distance calculations for matrix multiplication contributes to the significant efficiency gains.

Limitations and Future Work

  • Online Learning: The authors acknowledge that the separation of training and inference limits continuous online learning, which is crucial for adapting to evolving environments. They propose extending the quantization method to gradients and errors to enable on-device training and inference in the future.
  • Further Optimization: While significant improvements were shown, there's always room for further optimization in quantization techniques and fusion strategies.

Summary updated

Abstract

Deep convolutional neural networks (DCNNs) are currently popular in human activity recognition (HAR) applications. However, in the face of modern artificial intelligence sensor-based games, many research achievements cannot be practically applied on portable devices (i.e., smart phone, VR/AR). DCNNs are typically resource-intensive and too large to be deployed on portable devices, and thus, this limits the practical application of complex activity detection. In addition, since portable devices do not possess high-performance graphic processing units, there is hardly any improvement in Action Game (ACT) experience. Besides, in order to deal with multi-sensor collaboration, all previous HAR models typically treated the representations from different sensor signal sources equally. However, distinct types of activities should adopt different fusion strategies. In this paper, a novel scheme is proposed. This scheme is used to train 2-bit CNNs with weights and activations constrained to {-0.5, 0, 0.5}. It takes into account the correlation between different sensor signal sources and the activity types. This model, which we refer to as DFTerNet, aims at producing a more reliable inference and better trade-offs for practical applications. It is known that quantization of weights and activations can substantially reduce memory size and use more efficient bitwise operations to replace floating or matrix operations to achieve much faster calculation and lower power consumption. Our basic idea is to exploit quantization of weights and activations directly in pre-trained filter banks and adopt dynamic fusion strategies for different activity types. Experiments demonstrate that by using a dynamic fusion strategy, it is possible to exceed the baseline model performance by up to ~5% on activity recognition data sets, such as the OPPORTUNITY and PAMAP2 data sets. Using the quantization method proposed, we were able to achieve performances closer to that of the full-precision counterpart. These results were also verified using the UniMiB-SHAR data set. In addition, the proposed method can achieve ~9x acceleration on CPUs and ~11x memory saving.
BibTeX
@inproceedings{yang2018dfternet,
  author = {Zhan Yang and Osolo Ian Raymond and Chengyuan Zhang and Ying Wan and Jun Long},
  title = {{DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition}},
  booktitle = {IEEE Access},
  year = {2018},
  url = {https://openreview.net/forum?id=QVci5FWHz1},
}
Figure