MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition
Lists
Only you see this
Abstract
Multimodal sensors (e.g., visual, non-visual, and wearable) provide complementary information to develop robust perception systems for recognizing activities. However, most existing algorithms use dense sampling and heterogeneous sub-network to extract unimodal features and fuse them at the end of their framework, which causes data redundancy, lack of complementary multimodal information and high computational cost. In this paper, we propose a new novel multimodal neural architecture based on RGB and IMU wearable sensors (e.g., accelerometer, gyroscope) for human activity recognition called Multimodal Temporal Segment Attention Network (MMTSA). MMTSA first employs a multimodal data isomorphism mechanism based on Gramian Angular Field (GAF) and then applies a novel multimodal sparse sampling method to reduce redundancy. Moreover, we propose an inter-segment attention module in MMTSA to fuse multimodal features effectively and efficiently. We demonstrate the importance of imu data imaging and attention mechanism in human activity recognition by rigorous evaluation on three public datasets, and achieve superior improvements ($11.13\%$ on the MMAct dataset) than the previous state-of-the-art methods. The code is available at: https://github.com/THU-CS-PI/MMTSA.
BibTeX
@inproceedings{gao2023mmtsa,
author = {Ziqi Gao and Jianguo Chen and Junliang Xing and Shwetak N. Patel and Yuanchun Shi and Xin Liu and Yuntao Wang},
title = {{MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition}},
booktitle = {International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=vQXbQEDJi5},
}